Deep Learning For Natural Language Processing 2026

Term: Summer Semester

Time: Tuesdays, 13:30-17:00 AM

Course Overview

This course covers both foundation and up-to-date methodologies for Natural Language Processing (NLP), that today build the backbone of popular AI tools. Starting from the basics (mathematics of deep learning, backpropagation etc.), you will learn how more and more advanced models can understand natural language. By the end of this course, you will be gain understanding of:

Key machine learning paradigms and concepts
Both basic and advanced machine learning algorithms/models applied to NLP
Evaluation, optimisation, and comparison of NLP models
Applying NLP models and techniques to real-world problems

Prerequisites

Basic knowledge of linear algebra and calculus
Programming experience in Python
Probability and statistics fundamentals

Material

Lectures recordings are publicly available on YouTube.
Material (slides, exercises, homeworks, and exam solutions etc.) on Moodle.

Schedule

Week	Date	Topic
1	Apr 14	Intro to NLP + Mathematical foundations of Machine Learning Course Logistics Definition of text classification and generation task Evaluation Function minimization Efficient computation of gradients Stochastic Gradient Descent (SGD)
2	Apr 21	Log-linear models + Deep Neural Networks Fundamentals of machine learning problems Characteristics of the human language Fundamentals of vectors and linear functions Binary classification Tokenization Bag-Of-Words (BoW) and Byte-Pair-Econding (BPE) The sigmoid function Log-linear model (Logistic Regression) Loss functions and Binary Cross-Entropy Online and Minibatch SGD Multi-class classification Continuous Bag-Of-Words (CBOW) Softmax and Temperature Categorical Cross-Entropy Loss
3	Apr 6	Nonlinearity and Language Models Linearity and non-linearity in neural networks Anatomy of a neural network ReLU Fundamentals of probability Classic Language Model Markov chain property for word probability Maximum Likelihood Estimation Neural language models Decoding strategies Perplexity Dot product and cosine similarity Distributional Hypothesis Negative sampling Word2Vec (CBOW and Skip-gram variants) FastText Limits of cosine similarity
4	May 5	Recurrent Neural Networks RNN abstraction States and outputs Acceptor and Transducer RNNs Bidirectional RNN Elmann network Vanishing/exploding gradient Gates (hard and soft) LSTMs
5	May 12	Encoder-Decoder and Attention NLP “sequence” tasks (classification, labeling, generation) The issue of variable length generation PAD and EOS tokens as “dirty” solutions Encoder-Decoder architecture Teacher forcing Fundamentals of attention (formalization; explainability; generalization, calculation) Cross vs self attention
6	May 19	Transformers and BERT Motivation for the Transformer architecture Contextualization The encoder block and scaled dot-product Multi-head attention Residual connection Feed-Forward Network Positional embeddings Transfer learning BERT BERT pre-training (MLM + NSP) On LM development in academia vs industry Model complexity and explainability Finetuning Decoder heads in BERT Finetuning tasks Pretraining variants Pretrained LMs architectures
7	May 26	Decoder-only models and GPT Transformer architectures Attention masks Language modelling variants The decoder block Autoregressive decoder-only transformers (GPT) GPT-2 Zero-shot, one-shot, and few-shot learning GPT-3 and in-context-learning Zero-shot, one-shot and few-shot prompting Hallucinations Brief intro on reasoning and LLMs
8	Jun 2	Pushing boundaries of LLMs 1 Instruction tuning RLHF and DPO RAG and Langchain Tool-calling and agents Reasoning in LLMs Chain-Of-Thought prompting Self-consistency Limits of reasoning models Performance optimization in language models Mixed-precision pre-training Quantization KV-caching Adapters and LoRA Grokking
9	Jun 9	Pushing boundaries of LLMs 1 Motivations for Explainable AI Elements of Explainable AI Local VS global explanations Ante-hoc and post-hoc explanations Saliency VS Textual explanations Evaluating explanations Cognitive biases in humans Mixture of Experts models Dense vs Sparse Layers Routing and Top-K experts mechanism Fundamentals of Vision LLMs (LLAVA, CLIP) Fundamentals of Diffusion LLMs
10	Jun 30	Guest lecture (Cybersecurity, Interpretability, Mental health) Confidently Wrong: Why RL-Trained Models Still Ship Exploits ( Bhavyajeet Singh, UKP lab) Practical Bayesian Uncertainty for Language Models (Nico Daheim, UKP Lab) Natural Language Processing for Mental Health (Dr. Hiba Arnaout, UKP lab)
11	Jul 07	Exam simulation