NN 3

15 flashcards · Computer Science · by J M COOPER

1. Does having training data, a well designed NN and fitting method guarantee convergence to weights that give low error?

   No. A local minimum might be found – far from optimal, but hard for the fitting method to move away from Weights that lower error for some inputs might increase error for others – the fitting method oscillates There might simply be too much error – plenty of noise and not much signal

2. What is overfitting?

   Having too much convergence - we could have a perfect fit for the training data, but the idea is to make accurate predictions for new inputs So we want low generalisation error

3. How are training data and test data used?

   Can split the total data into 2 sets, training and test. Maybe 70/30 split Then learn weights using training data If reasonable convergence is achieved, calculate test error If training and test errors are similar and acceptably low, then good.

4. How does one design a neural network?

   Typically start with one hidden layer, num of hidden units set to (num of inputs + num of outputs) / 2 Try different configurations, retaining the best network (in terms of generalisation error) a number of experiments are required with each configuration to avoid being fooled if training locates a local minimum On each experiment, if under-learning occurs try adding more neurons to the hidden layer(s) if this doesn’t help, try adding an extra hidden layer If over-learning occurs, try removing…

5. What are neural networks used for?

   Medical phenomena Engine management Financial market prediction Jeopardy, Go, ...

6. How would we convert non numeric data for use in a NN? e.g. Gender = (Male, Female) Size = (S,M,L) Animal = (reptile, bird, mammal)

   Gender = (Male, Female) -> (0,1) Size = (S,M,L) -> (0,1,2) Animal = (reptile, bird, mammal) – use one-of-N encoding

7. How much data is needed for a NN?

   Standard heuristic n ≈ 10 × C where C is the number of connections no correct answer – depends on the nonlinear model depends on variance in noise, also unknown a priori

8. What does a priori mean?

   in a way based on theoretical deduction rather than empirical observation.

9. When is a NN not the right choice?

   If data is sparse or expensive

10. How might a small data set be leveraged?

    Ensemble & resampling methods

11. What issues are inherent in gathering data for NNs?

    changes in circumstance invalidate historic data Data has to cover all eventualities – a NN trained on low incomes will predict nothing about high incomes Easiest features are learned first – even if they aren’t the features of interest Unbalanced data may not be applicable - ege a NN trained on general population data should not be used to make predictions about visitors to a clinic (disease rates higher in latter)

12. When are NNs a good choice?

    Lots of cheap, numeric data in reasonable ranges Not too many outliers Don’t know or don’t care about the model relating input to response Happy to wait a long time for a predictor Don’t care about interpretability

13. What is deep learning?

    Just very deep/complex NNs

14. WHy should NNs be as simple as possible?

    The computational complexity of learning the weights increases with increasing node/layer complexity More chance of overfit if too complex Parsimony – the philosophical idea that the simple solutions should be preferred to complex ones

15. How has the deep learning community overcome issues associated with complex NNs?

    Address complexity and (hopefully) overfit: omit some nodes and/or connections input a random amount of zero data during learning use high-performance computing use GPUs for fast vector and matrix calculations Long Short-Term Memory – some units with the identity activation function, and self connection with fixed weight of 1 backpropagated errors can’t vanish or explode due to the constant derivative Ignore the parsimony philosophy – just find something that beats SVM, random forests, etc.