| 1 | Apr 22 | NLP tasks and evaluation - Course Logistics
- Definition of text classification and generation task
- Evaluation
|
| 2 | Apr 29 | Mathematical foundations of deep learning - Function minimization
- Efficient computation of gradients
- Stochastic Gradient Descent (SGD)
|
| 3 | May 6 | Log-linear models - Ambiguity of human language
- Fundamentals of vectors and linear functions
- Binary classification
- Tokenization
- Bag-Of-Words (BoW) and Byte-Pair-Econding (BPE)
- Nonlinear mapping and sigmoid function
- Log-linear model (Logistic Regression)
|
| 4 | May 13 | Deep Neural Networks - Loss functions and Binary Cross-Entropy
- Online and Minibatch SGD Multi-class classification
- Continuous Bag-Of-Words (CBOW)
- Softmax and Temperature
- Categorical Cross-Entropy Loss
- Linearity and non-linearity in neural networks
- ReLU
|
| 5 | May 20 | Language models and word embeddings - Fundamentals of probability
- Classic Language Model
- Markov chain property for word probability
- Maximum Likelihood Estimation
- Perplexity
- Neural language models
- Word vector lookup
- Basic text generation with neural LM
- Greedy decoding and sampling
|
| 6 | May 27 | Learning Word Embeddings - Dot product among vectors and cosine similarity
- Distributional Hypothesis
- Negative sampling
- Word2Vec (CBOW and Skip-gram variants)
- FastText
- Limits of cosine similarity
|
| 7 | Jun 3 | Recurrent neural networks (RNNs) - RNN abstraction
- States and outputs
- Acceptor and encoder RNNs
- Bidirectional RNN
- Simple RNN (Elmann network)
- Vanishing/exploding gradient
- Gates (hard and soft)
- LSTMs (input, keep, and forget gates; solving the vanishing gradient)
|
| 8 | Jun 10 | Autoregressive encoder-decoder with RNNs and attention - NLP “sequence” tasks (classification, labeling, generation)
- The issue of variable length generation
- PAD and EOS tokens as “dirty” solutions
- Encoder-Decoder architecture
- Teacher forcing
- Fundamentals of attention (formalization; explainability; generalization, calculation)
- Cross vs self attention
|
| 9 | Jun 24 | Transformers, Self-attention and BERT (double lecture) - Motivation for the Transformer architecture
- Contextualized text representations
- The encoder block
- Scaled dot-product
- Multi-head attention
- Parallelizing MHA
- Residual connection
- Positional embeddings
- Encoder vs Decoder block
- Transfer learning
- BERT
- Pretraining objectives of BERT (MLM, NSP)
- On LM development in academia vs industry
- Model complexity and explainability
- Finetuning
- Decoder heads in BERT
- Finetuning tasks
- Pretraining variants
- Pretrained LMs architectures
|
| 10 | Jul 1 | Decoder-only Models and GPT - Types of Transformers (encoder-decoder, encoder-only, decoder-only)
- Attention masks
- Full, prefix and masked language modelling
- Autoregressive decoder-only transformers (GPT-2)
- Zero-shot, one-shot, and few-shot learning
- In-context-learning
- Zero-shot, one-shot and few-shot prompting
- Hallucinations
- Brief intro on reasoning and LLMs
- Data contamination
- Continuous prompts
|
| 11 | Jul 8 | Contemporary LLMs and Explainability - Instruction tuning
- RLHF
- Toolformer
- Motivations for Explainable AI
- Elements of Explainable AI
- Local VS global explanations
- Ante-hoc and post-hoc explanations
- Saliency VS Textual explanations
- Evaluating explanations
- Cognitive biases in humans
- Wrong agreement
- Likeability-Effectiveness tradeoff
- Fairwashing
- Manipulation
- bouncer problem
- Antropomorphization of AI
|
| 12 | Jul 15 | Exam simulation |
| 13 | Jul 22 | Guest lecture (Mechanistic interpretability, Privacy and Security in LLMs, Culture-aware LLMs) - Do Language Models Dream of Electric Clocks? (Federico Tiblias, UKP lab)
- Privacy and Security in Large Language Models (Anmol Goel, UKP Lab)
- Global Voices, Responsible Models; Cultural Evaluation and Adaptation in LLMs (Cecilia Liu, UKP lab)
|