Goldsmiths: GPT

~Attention defines a network on the fly

8 flashcards · Computer Science · by lakesare

Where was the Transformer deep learning model introduced?
In the 2017 Google paper "Attention Is All You Need". Transformers are used primarily in the field of natural language processing (NLP).
What's the difference between transformers and recurrent neural networks (RNNs)?
Both RNNs and Transformers are designed to handle sequential data, buttransformers do not require the sequential data to be processed in order. For example, if the input data is a natural language sentence, the Transformer does not need to process the beginning of it before the end. Due to this feature, the Transformer allows for much more parallelization than RNNs and therefore reduced training times.
Name 2 pretrained Transformer systems.
BERT (Bidirectional Encoder Representations from Transformers) GPT (Generative Pre-trained Transformer), Since the Transformer model facilitates more parallelization during training, it has enabled training on larger datasets than was possible before it was introduced. This has led to the development of pretrained systems such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer), which have been trained with huge general language dataset…
What's a neural network Jacobian?
Sensitivity of the neural network outputs with respect to inputs. Mathematically it's just a matrix of partial derivatives. Jacobian lets us understand what neural network is focused on.
What's the input & output of RNN (recurrent neural network)?
RNNs take sequences of inputs, and produce sequences of outputs.
What's a softmax function?
It's a generalisation of the logistic function to multiple dimensions. Logistic function
Hard attention VS soft attention?
Hard attention - we direct the fixed-size camera. Soft attention - other regions are kind of visible too.
What does GPT stand for?
Generative Pretrained Transformer