Stanford CS224N

Natural Language Processing (NLP) is a field at the intersection of computer science, artificial intelligence, and linguistics. The goal is to enable computers to understand, interpret, and generate human language in a valuable way.

• Understanding text and speech
• Machine translation
• Question answering systems
• Sentiment analysis
• Information extraction
• Text generation and summarization

Human language is incredibly complex and nuanced. Several factors make NLP particularly challenging:

• Ambiguity: Words and sentences can have multiple meanings depending on context
• Context Dependence: Meaning changes based on surrounding words and discourse
• World Knowledge: Understanding often requires external knowledge beyond the text
• Variability: Infinite ways to express the same idea
• Compositionality: Meaning of phrases depends on word combinations

Traditional approaches used discrete symbols (one-hot encoding), but this fails to capture semantic relationships. Modern approaches use distributed representations.

Word2Vec is a framework for learning word embeddings using neural networks. Two main architectures:

• Skip-gram: Predicts context words given a center word
• CBOW (Continuous Bag of Words): Predicts center word given context words

Word vectors capture semantic and syntactic relationships through vector arithmetic:

Training word vectors requires optimizing the objective function using gradient descent.

• Gradient descent updates parameters in direction of steepest descent
• Stochastic Gradient Descent (SGD) uses mini-batches for efficiency
• Learning rate controls step size
• Negative sampling reduces computational complexity

Efficient training techniques make Word2Vec practical for large corpora:

• Negative Sampling: Sample k negative examples instead of full softmax
• Hierarchical Softmax: Use binary tree structure for efficient computation
• Subsampling: Reduce frequency of common words
• Context Window: Dynamic window size for varied context

GloVe combines global matrix factorization and local context window methods.

• Leverages global word-word co-occurrence statistics
• Fast training and good performance with smaller corpora
• Captures both semantic and syntactic patterns

Two main approaches to evaluate word embeddings:

• Intrinsic Evaluation: Word similarity, analogy tasks (e.g., king:queen :: man:?)
• Extrinsic Evaluation: Performance on downstream tasks (NER, sentiment analysis)
• Intrinsic is faster but may not correlate with real-world performance
• Extrinsic is more meaningful but computationally expensive

Language models assign probabilities to sequences of words.

• Core task: predict next word given previous words
• Applications: speech recognition, machine translation, text generation
• Perplexity used as evaluation metric

Comparison of traditional and neural approaches:

• N-gram Models: Use count-based statistics with Markov assumption
• Neural Models: Learn distributed representations, better generalization
• N-grams suffer from data sparsity and curse of dimensionality
• Neural models handle longer contexts and unseen word combinations

Neural networks are composed of layers of interconnected neurons that transform inputs to outputs.

• Input Layer: Receives raw data (e.g., word vectors)
• Hidden Layers: Learn hierarchical representations
• Output Layer: Produces final predictions
• Activation Functions: Introduce non-linearity (sigmoid, tanh, ReLU)

Forward propagation computes the output of the network given an input.

• Data flows from input to output layer
• Each layer applies linear transformation followed by non-linearity
• Final layer often uses softmax for classification

Backpropagation efficiently computes gradients using the chain rule of calculus.

• Compute loss at output layer
• Propagate error backwards through network
• Calculate gradient for each parameter
• Update parameters using gradient descent

Computing gradients efficiently is crucial for training deep networks.

• Local Gradients: Derivative of operation with respect to inputs
• Upstream Gradients: Gradient flowing from layers above
• Downstream Gradients: Product of local and upstream gradients
• Computational graph helps visualize gradient flow

Matrix calculus enables efficient computation of gradients for vectorized operations.

• Jacobian matrix contains all partial derivatives
• Chain rule extends to matrix operations
• Vectorization improves computational efficiency

Advanced optimization methods improve training speed and convergence.

• SGD: Stochastic Gradient Descent with mini-batches
• Momentum: Accumulates velocity to accelerate convergence
• Adam: Adaptive learning rates per parameter
• Learning Rate Scheduling: Decay learning rate over time

Dependency grammar represents sentence structure as directed graphs showing word relationships.

• Words are nodes, dependencies are directed edges
• Each word has exactly one head (except root)
• Captures grammatical relations (subject, object, modifier)
• More flexible than phrase structure grammar
• Works well across different languages

Common dependency relations capture grammatical functions:

• nsubj: Nominal subject
• dobj: Direct object
• amod: Adjectival modifier
• prep: Prepositional modifier
• det: Determiner
• aux: Auxiliary verb

Example: "The cat sat on the mat"
sat (root) → cat (nsubj), on (prep) → mat (pobj), cat → The (det), mat → the (det)

Transition-based parsers use a sequence of actions to build dependency trees.

• Configuration: (Stack, Buffer, Dependencies)
• SHIFT: Move word from buffer to stack
• LEFT-ARC: Add dependency from top to second-top, remove second-top
• RIGHT-ARC: Add dependency from second-top to top, remove top

Neural networks learn to predict parsing actions from parser configurations.

• Extract features from stack, buffer, and existing dependencies
• Use word embeddings, POS embeddings, dependency label embeddings
• Feed-forward network predicts next action
• Much faster and more accurate than traditional parsers

Parser performance measured by attachment scores:

• UAS (Unlabeled Attachment Score): % of words with correct head
• LAS (Labeled Attachment Score): % of words with correct head and label
• LAS is stricter and more informative
• Modern neural parsers achieve >95% UAS on English

Recurrent Neural Networks process sequential data by maintaining hidden state across time steps.

• Handle variable-length sequences
• Share parameters across time steps
• Maintain memory of previous inputs
• Natural fit for language modeling, machine translation, speech

Training RNNs on long sequences faces gradient instability issues.

• Vanishing Gradients: Gradients shrink exponentially, preventing learning of long-term dependencies
• Exploding Gradients: Gradients grow exponentially, causing numerical instability
• Caused by repeated multiplication of weight matrix
• Solutions: gradient clipping, better architectures (LSTM, GRU)

LSTMs use gating mechanisms to control information flow and maintain long-term memory.

• Forget Gate: Decides what information to discard from cell state
• Input Gate: Controls what new information to add
• Output Gate: Determines what to output based on cell state
• Cell state acts as highway for gradient flow

GRUs simplify LSTM architecture while maintaining similar performance.

• Combines forget and input gates into single update gate
• Merges cell state and hidden state
• Fewer parameters than LSTM
• Often performs comparably to LSTM with faster training

Bidirectional RNNs process sequences in both forward and backward directions.

• Forward RNN processes left-to-right
• Backward RNN processes right-to-left
• Concatenate forward and backward hidden states
• Captures context from both past and future
• Useful for tasks where full sequence is available (e.g., NER, POS tagging)

RNNs and their variants power many NLP applications:

• Language Modeling: Predict next word in sequence
• Machine Translation: Encoder-decoder architectures
• Sentiment Analysis: Classify text sentiment
• Named Entity Recognition: Tag entities in text
• Speech Recognition: Convert audio to text
• Text Generation: Generate coherent text sequences