Machine Learning: An In-depth Visual Introduction

Make Your Own Neural Network in Python: A Simple Guide on Machine Learning with Neural Networks, by Michael Taylor.

18 flashcards · Computer Science · by lakesare

What are required & optional hyperparameters? In the previous chapters, we touched on a variety of neural network architectures such as CNNs and RNNs. Although each is slightly different, all have a structure - or shell - that is made up of similar parts. These parts are called hyperparameters, and include elements such as the number of layers, nodes and the learning rate.
Required Hyperparameters Include: Total number of input nodes Total number of hidden layers Total number of hidden nodes in each hidden layer Total number of output nodes Weight values Bias values Learning Rate Optional Hyperparameters Include: (This list is not an exhaustive list.) Learning rate schedule (a.k.a learning rate decay) Momentum Mini-batch size Weight decay Dropout
For example, if you have an image of 16x16 pixels as input, you have a total of 256 input nodes. This is because each input node represents a single image pixel, and there are 256 pixels in total (16x16). What are 2 other names for these 256 nodes of the input vector?
256 nodes 256 features 256 dimensions
What can happen if learning rate is too low learning rate is too high
too low - neural network can take too long to converge. too high - can fail to reach a global minimum due to overshooting and/or diverging.
What's momentum in a neural network? Optional Hyperparameters Include: Learning rate schedule (a.k.a learning rate decay) Momentum Mini-batch size Weight decay Dropout
Momentum is a value that is used to help push a network out of a local minimum, which is a false "lowest-error" that networks often become trapped in. Technically, momentum must be predetermined for the network, and is also one of the most popular techniques added to backpropagation.
Here's a formula for the sigmoid activation function. What's x?
x is a net input to a particular node (all inputs passed through the summation function!)
What's the relationship between logistic (sigmoid) and tanh (hyperbolic tangent) functions?
Sigmoid function is the translated and scaled version of tanh!
Name 2 reasons why logistic (== sigmoid) and tanh (== hyperbolic tangent) functions are so popular as activation functions.
They introduce non-linearity - activation functions such as the tanh and logistic essentially "break" the linearity of a network and enable it to solve more complex problems. They limit the output to a certain range - tanh() produces output between -1 and 1, sigmoid function produces output between 0 and 1.
What's uppercase and lowercase for sigma letter?
uppercase Σ lowercase σ
What's a cost function?
Returns a total error of the network. Within a neural network, a cost function transforms everything that occurs within the network into a number that represents the total error of the network.
Why is the cost function often referred to as an objective function?
Because the objective of the network is to minimize the cost function.
Name 2 examples of cost functions.
Mean Squared Error Squared Error Root Mean Square Error Sum of Square Errors
What's n, ti and zi in the Mean Squared Error cost function.
n - total number of output nodes ti - desired (target) output of the particular output node zi - actual output of the particular output node The Mean Squared Error function takes the sum of all squared output errors in a network and averages them. In other words, the MSE measures the difference between the target output and actual output of training examples in a network.
What's a partial derivative?
A partial derivative is the derivative of a function which has two or more variables but with respect to only one variable. What's more, all the other variables are treated as constant. In other words, a partial derivative enables you to measure how a single variable (out of many) impacts another single variable.
In relation to neural networks - how are partial derivatives helpful?
Allows to discover how a change in weight w1 affects the total error - while all the other weights remain constant. The same can be calculated for w2, w3, bw1, bw2 etc.
What does this (likely) mean?
This is a partial derivative of the error function (E) with respect to weight 5 (W5).
What's n, xi and wi in summation operator? Every single hidden and output node contains the summation operator.
n - number of inputs to that node. xi - output of every input node i wi - weight of every input node i
Our answer is gradient descent, which is an optimization method that helps us find the exact combination of weights and ultimately discover a global minimum. This, however, raises a third and final question: how do we perform gradient descent?
To perform gradient descent we must calculate the partial derivatives of each weight, and update each weight.
What’s more, the partial derivatives for output layer weights are always calculated first because their values are used to calculate the interior partial derivatives. Let’s apply the chain rule and unpack our first derivative. On a high level, working backwards from the total error, we can see that there are two intermediate variables to be concerned with. First, we need to know how much a change in the output z affects the total error. Second, we need to know how much a change in weight w5 aff…