Speech and Language Processing (Jurafsky)
An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Daniel Jurafsky, Stanford University.
41 flashcards · Computer Science · by lakesare
Lemma VS wordform?
Wordforms: run, runs, ran and running Their lemma: run.
[Philosophy] Type VS token?
The sentence "A rose is a rose is a rose" contains 3 types, "a", "rose", and "is", and and 8 tokens of those types, "a", "rose", "is", "a", "rose", "is", "a", "rose".
[Text normalization] What's text tokenization? Tasks are commonly applied as part of any normalization process:Tokenizing words Normalizing word formats Segmenting sentences
Tokenization divides text into SEPARATE words. Depending on the application, tokenization algorithms may:- tokenize words (defined by spaces in orthographies that have spaces, or more complex algorithms),- tokenize characters (as in Chinese, 姚 Yao 明 Ming 进 enter 入 enter 总 overall 决 decision 赛 game),- tokenize subwords (if our training corpus contains the words low, and lowest, but then the word lower appears in our test corpus, our system will know what to do with it).
What's the probability of all these words fitting together?
What's the Markov assumption?
It's the assumption that we can calculate the probability of
What's the probability that I follows the beginning of the sentence P(I|<s>)?
2/3. Count of the times <s>I occurs, divided by the count of the times <s> occurs.
What's holdout data?
We divide our data into training set test set (this is the holdout data) Hold-out is when you split up your dataset into a 'training' and 'test' set. The training set is what the model is trained on, and the test set is used to see how well that model performs on unseen data.
What's the multiple discovery hypothesis?
Hypothesis that most scientific discoveries and inventions are made independently and simultaneously by multiple scientists and inventors. Commonly cited examples of multiple independent discovery are the 17th-century independent formulation of calculus by Isaac Newton, Gottfried Wilhelm Leibniz and others, described by A. Rupert Hall
Name 2 widespread anchors in regexes.
/^hi/ - caret, matches the beginning of the string. /hi$/ - dollar sign, matches the end of the string. Anchors are special characters that anchor regular expressions to particular place in a string.
[Finite-state automaton (FSA)] How can we augment this display of the FSA for further clarity? A finite-state automaton for talking sheep (baa! baaa!).
Add the fail state! The algorithm will fail whenever there is no legal transition for a given combination of state and input. The input abc will fail to be recognized since there is no legal transition out of state q0 on the input a.We can think of these "empty" elements in the table as if they all pointed at one "empty" state, which we might call Fail state the fail state or sink state.
[Finite-state automaton (FSA)] Name 2 ways to come up with nondeterministic finite-state automaton.
getNextState(currentState, letter) => can return multiple states! When we get to state 2, if we see an a we don’t know whether to remain in state 2 or go on to state 3. ε-transitions If we are in state 3, we are allowed to move to state 2 without looking at the input, or advancing our input pointer. So this introduces another kind of non-determinism - we don't know whether to follow the ε-transition VS the ! arc. Compare with the identical deterministic finite-state automaton.
[Finite-state automaton (FSA)] Name 3 solutions to the problem of nondeterminism.
Backup: Whenever we come to a choice point, we could put a marker to mark where we were in the input, and what state the automaton was in. Then if it turns out that we took the wrong choice, we could back up and try another path. Look-ahead: We could look ahead in the input to help us decide which path to take. Parallelism: Whenever we come to a choice point, we could look at every alternative path in parallel.
His family is blithely unaware of his nature.
In a way that shows a casual and cheerful indifference considered to be callous or improper.
[Finite-state automaton (FSA)] How would you present FSA as a tape of inputs? Non-deterministic FSA, where getNextState(currentState, letter) => can return multiple states! When we get to state 2, if we see an a we don't know whether to remain in state 2 or go on to state 3.
[Finite-state automaton (FSA)] What's the relationship between deterministic and non-deterministic FSAs?
There's an algorithm for converting an NFSA to DFSA, but the number of states in this deterministic FSA may be much larger. For any NFSA there is an equivalent FSA.
What's a clitic?
A clitic is a morpheme that acts as a word, but treated phonetically as forming a part of a neighboring word. English: they'reFrench: l'opera. Note that orthography is not always a good guide for distinguishing clitics from affixes: clitics may be written as separate words, but sometimes they are joined to the word they depend on (like the Latin clitic -que, meaning "and"), or separated by special characters such as hyphens or apostrophes (like the English clitic 's in "it's").
What are modal verbs?
A modal verb is a type of verb that is used to indicate modality – that is: likelihood, ability, request, capacity, suggestions, order, obligation, or advice. In English, the modal verbs commonly used are can, could, must, should, had better, have to and sometimes need or dare. Modal verbs always accompany the base (infinitive) form of another verb having semantic content. Jurafsky book distinguishes between: Main verbs (eat, sleep, impeach), modal verbs (can, will, should), and primary verbs […
Proclitics VS enclitics? Recall that a clitic is a unit whose status lies in between that of an affix and a word.
proclitics - appears before its host (l'opera) enclitics - appears after its host (they're)
What's concatenative vs nonconcatenative morphology?
concatenative morphology - word is composed by concatenating morphemes (unbreakable) nonconcatenative morphology - root is modified (freeze → froze).
What's a gloss?
Marginal notation explaining the main text in a document. Shown is a parchment page from the Royal Library of Copenhagen. Mid 16th century alteration of the noun gloze, from Old French glose (see gloze), suggested by medieval Latin glossa ‘explanation of a difficult word’, from Greek glōssa ‘word needing explanation, language, tongue’.
What's a lexicon?
A lexicon is an explicit list of every word of the language. Every word, i.e., including abbreviations and proper names: a, AAA, AA, Aaron, aback, ~.
What's morphotactics?
Morpheme ordering restrictions. We can do doability - do + -able + -ity but not *doityable do + -ity + -able. The suffix -ity produces a noun from an adjective, and -able creates adjectivess from verbs. To reverse the order violates the rules of English morphotactics, making the word ungrammatical.
Inflection VS derivation?
inflection - changes tense/mood/gender/~. dog ➝ dogs, wait ➝ waited.Inflection is the process of adding inflectional morphemes that modify a verb's tense, mood, aspect, voice, person, or number or a noun's case, gender, or number, rarely affecting the word's meaning or class.derivation - changes either meaning or the part of speech. write ➝ rewrite, slow ➝ slownessDerivation is the process of adding derivational morphemes, which create a new word from existing words and change the semantic mean…
What's ablaut (or apophony)?
A sound change that that indicates grammatical information (often inflectional). sing, sang, sung, song bind, bound blood, bleed brood, breed doom, deem food, feed lie, lay rise, rose, risen weave, wove foot, feet goose, geese tooth, teeth
What's a gerund?
A gerund is the "-ing" form of a verb when it functions grammatically as a noun in a sentence. E.g. 'Swimming is an excellent form of exercise!' Gerunds are identical in appearance to present participles, but they are not used to form tenses of the verb or provide adjectival information.
Name 2 noun inflections. English has 9 inflectional morphemes, 2 of which are noun inflections.
-s/-es (plural) 's/'/s (genitive) English has only 9 inflectional morphemes:
[Finite-state automaton (FSA)] Explain what this FSA tries to show? A finite-state automaton for English nominal inflection.
This FSA checks whether some English noun is plural-inflected well. We start from nothing (no noun, nothing), and EITHER OR: add regular noun + plural -s, or These are the vast majority of English nouns since for now we will ignore the fact that the plural of words like fox have an inserted e: foxes.irregular plural noun, or irregular singular noun.
What are the capital letters of σ (sigma) and δ (delta)?
Sigma σ: Σ Delta δ: Δ
[Finite-state automaton (FSA)] What do acceptor finite-state automatons do? Acceptors are also called recognizers. Finite-state automatons can be subdivided into acceptors, classifiers, transducers, and sequencers.
=> return true/false (whether received input was accepted) Acceptors (also called detectors or recognizers) produce binary output, indicating whether or not the received input is accepted. Each state of an acceptor is either accepting or non accepting. Once all input has been received, if the current state is an accepting state, the input is accepted; otherwise it is rejected. As a rule, input is a sequence of symbols (characters); actions are not used. The start state can also be an accepting…
[Finite-state automaton (FSA)] What do transducer finite-state automatons do? Finite-state automatons can be subdivided into acceptors, classifiers, transducers, and sequencers.
Transducer FSA does everything that acceptor FSA does - except it also outputs a string.
[Distributional hypothesis] What's distributional hypothesis?
Idea that words that occur in similar contexts tend to have similar meanings. The distributional hypothesis suggests that the more semantically similar two words are, the more distributionally similar they will be in turn, and thus the more that they will tend to occur in similar linguistic contexts.
Semantic similarity VS semantic relatedness?
semantic similarity - synonyms or nearly synonyms (cat, dog) Semantic similarity is a metric defined over a set of documents or terms, where the idea of distance between items is based on the likeness of their meaning or semantic content as opposed to lexicographical similarity. semantic relatedness - associatively connected (coffee, cup) Semantic relatedness includes any relation between two terms, while semantic similarity only includes "is a" relations.Consider the meanings of the words coff…
[Distributional hypothesis] What does the entry '60' indicate? A co-occurrence matrix
It indicates how many times the row word 'strawberry' is surrounded (in the ±4 word window) by the column word 'pie'. Note that the co-occurrence matrix is always symmetric - the entry with the row word 'pie' and the column word 'digital' will be 5 as well (as these words co-occur in the very same sentences!).
[Distributional hypothesis] What does tf-idf stand for?
Term frequency – inverse document frequency
[Distributional hypothesis] Giving this co-occurrence matrix - what's the cosine similarity between cherry and information cos(cherry, information)? A (rough) graphical demonstration of cosine similarity, showing vectors for three words (cherry, digital, and information) in the two dimensional space defined by counts of the words computer and pie nearby. Note that the angle between digital and information is smaller than the angle between cherry and information.
[Distributional hypothesis] What are word embeddings?
Word embeddings are short dense vector representations of words. Word embeddings are short dense vectors. Unlike the vectors we’ve seen so far, embeddings are short, with number of dimensions d ranging from 50-1000, rather than the much larger vocabulary size |V| or number of documents D we’ve seen. These d dimensions don’t have a clear interpretation. And the vectors are dense: instead of vector entries being sparse, mostly-zero counts or functions of counts, the values will be real-valued num…
[Distributional hypothesis] What's the intuition behind word2vec?
We make neural network create weights for 'Is word w likely to show up near apricot?'. Instead of counting how often each word w occurs near, say, apricot, we’ll instead train a classifier on a binary prediction task: “Is word w likely to show up near apricot?” We don’t actually care about this prediction task; instead we’ll take the learned classifier weights as the word embeddings. Treat the target word and a neighboring context word as positive examples. Randomly sample other words in the le…
[Distributional hypothesis] What words would be similar in the word2vec model, or raw co-occurrence model?
Words that have very similar contexts. If two different words have very similar “contexts” (that is, what words are likely to appear around them), then our model needs to output very similar results for these two words.And what does it mean for two words to have similar contexts? I think you could expect that synonyms like “intelligent” and “smart” would have very similar contexts. Or that words that are related, like “engine” and “transmission”, would probably have similar contexts as well.
[Context-free grammar] Terminal VS non-terminal symbols?
terminal symbols - words in the language. non-terminal symbols - express abstractions over these terminals. The symbols terminal that correspond to words in the language ("the", "nightclub") are called terminal symbols; the lexicon is the set of rules that introduce these terminal symbols.The non-terminal symbols that express abstractions over these terminals are called non-terminals.
[Context-free grammar] Why is the use of formal languages to model natural languages called generative grammar?
Because we define the formal language as the set of all possible sentences generated by the grammar. In linguistics, the use of formal languages to model natural languages is called generative grammar since the language is defined by the set of possible sentences "generated" by the grammar.
What's a treebank?
In linguistics, a treebank is a parsed text corpus that annotates syntactic or semantic sentence structure. The Penn Treebank Penn Treebank project (whose POS tagset we introduced in Chapter 8) has produced treebanks from the Brown, Switchboard, ATIS, and Wall Street Journal corpora of English, as well as treebanks in Arabic and Chinese.