B031278 - Deep Learning. Fall 2024

Instructor

Paolo Frasconi, DINFO, via di S. Marta 3, 50139 Firenze

email: .

Office Hours

Tuesday Monday 10:45-12:45.

Description

The course aims to provide an overview of classic and some current deep learning methodologies. It will tentatively cover the following aspects:

• Fundamentals: Supervised and unsupervised learning problems. Supervised learning in single-layer networks. Loss functions, optimizers. Deep networks. computational graphs, expressiveness, regularizers, optimizers, normalizers.
• Deep architectures: convolutional, recurrent, attention and transformers, graph neural networks.
• Software frameworks: Tensor operations, automatic differentiation, manipulating data, implementing deep architectures (see blended teaching below).
• Beyond single task: domain adaptation and generalization, multi-task learning, transfer learning
• Theory: optimization and generalization in overparameterized systems.
• Empirical deep learning: experimental design and reproducibility, hyperparameter optimization, neural architecture search.
• Unsupervised deep learning: Autoencoders, anomaly detection, feature extraction, self-supervised learning.

Blended teaching

Starting this year, I will experiment a blended form of teaching with 6 online hours (less than 1 credit) consisting of six 30-minutes video-lectures, mainly covering practical deep learning programming aspects, with a focus on PyTorch and Tensorflow. There will be a question answering session in class, one week after each video-lecture is posted. Please email me in advance at with the questions that you would like to discuss in class.

Learning objectives

You will be able to understand and apply state-of-the-art algorithms and architectures, to understand the relevant methodological details, and to operate according to the current practices. Deep learning is a fast moving field. To be successful in your future career you will need to develop sufficient skills to competently read and understand a large fraction of the current and future literature (yes, a form of meta-learning). Thus, after succesfully completing this course, you should be able to understand, reimplement and evaluate on your own many novel algorithms, with limited help or guidance from a supervisor.

Prerequisites

Multivariate calculus and linear algebra are much needed. Elementary numerical optimization, algorithms and data structures, and proficiency on scientific computing with a modern programming language (e.g., NumPy with Python) will be useful.

Suggested readings

Starting this year, I will try to use textbook material whenever possible. There are two reasons for this change. First, the course has been moved to the very beginning of the AI program and many of you are still not comfortable studying from papers. Second, two very useful and up-to-date books have been published this year. Still, some material is better covered by reading relevant papers.

Assessment

There is a single oral final exam with an associated project. You can choose topic of your project but you should discuss it with me during office hours and I will give you the details of what should be done.

Typically, you will be assigned one or more papers to read and will be asked to work at home to reproduce some (simplified) experimental results or to apply the same method to different data or in a slightly different setting. You are responsible for studying the relevant methodological and theoretical prerequisites of these papers (in some cases, studing the references covered in class may be sufficient but in other cases, especially when dealing with the details of the experimental procedures, readings other ancestors in the citation graph may be necessary).

There is no need to submit a report for your work, but you are asked to share with me the code (not the data! --- for that a links is sufficient) you have developed with some short instructions for reproducing your results. Small zip files can be shared by email (please send a https://0x0.st/ link if the zip is over a MByte) but if you prefer to share a git repository, please create a private one on https://codeberg.org/ and share it with me by inviting the user dl-unifi as a member.

You will be required to give a short presentation during the exam. Please ensure that during your presentation you introduce and motivate the problem being addressed in the context of the relevant literature, explain the technical derivation of the methods, and describe in detail the experimental work and the results. You are allowed (but not required) to use multimedia tools to prepare your presentation. You should be prepared to answer general questions about the background literature supporting your paper(s) (for example if the method uses an optimizer, which happens with overwhelming probability, you are supposed to know how it works) and about the details of your experimental work.

You can work in groups of two to carry out the experimental work (three is an exceptional number that you must motivate clearly). If you do so, please ensure that personal contributions to the overall work are clearly identifiable. In any case, during the exam you will have to answer questions individually.

Schedule and reading materials

Relevant papers and/or sections of the textbook(s) are listed on the right side. [Sections of] papers in the "required" list have been covered in class and should be studied while preparing for the exam. Papers listed as "optional" may be useful to get a better picture of the class topic but you do not need to study them, unless they are directly related to the topic of your project.

Date	Topics	Readings/Handouts
2024-09-17	Administrivia. Most common forms of learning: supervised, unsupervised, reinforcement. Outline of the course. Historical remarks.	BB24 Chapter 1, GBC16 5.1 Optional additional readings: (Schmidhuber 2015);
2024-09-20	Supervised learning and empirical risk minimization. Optimal (binary) classifier. Modeling: the generative and the discriminative direction. Single layer networks. Role of the logistic function.	BB24 4.1, 4.2, 5.2, 5.3, GBC16 5.5. Optional additional readings: (Ng & Jordan 2001);
2024-09-24	Setting up deep learning frameworks. Working remotely. Tensors.	ZLLS23 2.1-2.3 Required additional readings: (Video lecture 1);
2024-09-24	"Linking MLE and ERM. Loss functions for regression: square, Huber. The least squares problem and its solution. Loss function for classification: 0-1, hinge, log, exp. Generalized linear models: Bernoulli and binary classification."	BB24 4.1, 5.3, GBC16 5.5, 5.7. P24 5.1-5.4. Optional additional readings: (Girshick 2015); (Huber 1963);
2024-09-27	"Logistic regression as a special case of generalized linear model. Multiclass classification and softmax regression. Gradient descent."	BB24 5.4, 7.1, 7.2. P24 5.5, 5.6, 5.7. Optional additional readings: (Shalev-Shwartz & Ben-David 2014);
2024-10-01	Logistic regression in pure Tensorflow. Tensorboard. Logistic regression in PyTorch	ZLLS23 4 Required additional readings: (Video lecture 2); (Video lecture 2 src);
2024-10-01	Stochastic gradient descent. Convergence rates of GD and SGD. The tradeoffs of large scale learning. Optimization, estimation, and approximation errors.	BB24 7.1, 7.2; GBC16 8.1; ZLLS23, 12.3, 12.4 Required additional readings: (Bottou & Bousquet, 2007); Optional additional readings: (Bottou et al. 2018);
2024-10-04	Minibatches. Effects of batch size. SGD with momentum. Adagrad. RMSProp.	BB24 7.2, 7.3; ZLLS23 12.4-12.8 Optional additional readings: (Shallue et al. 2019); (Duchi et al. 2011); (Hinton 2012);
2024-10-08	Basis function. RBF networks. Feature engineering and its limitations. Biologically inspired features. Sparse coding. Feature learning and end-to-end learning. Compositionality and deep representations.	GBC16 1, 13.4 Optional: BB24 4.1.1, 16.1. Required additional readings: (Bengio et al. 2013); Optional additional readings: (Lee et al. 2009); (Ringach 2002); (Olshausen & Field, 1996); (Hinton & Salakhutdinov 2006); (Poggio & Girosi 1990); (Serre et al. 2005);
2024-10-11	Layerwise training of deep networks. Denoising autoencoders. Multilayered perceptrons. Rectifiers.	BB24 6.1-6.3, GBC16 6, 6.1, 6.3, 6.4, 14.2, P24 4.1-4.3 Optional additional readings: (Hinton et al. 2006); (Hinton & Salakhutdinov 2006); (Vincent et al. 2008); (Glorot et al. 2011); (Nair & Hinton 2010);
2024-10-15	More activation functions: Softplus, Leaky ReLU, parametric ReLU. MLPs of ReLUs are linear piecewise functions. Inductive bias. Automatic differentiation in forward and reverse mode.	BB24 8, 9.1 Required additional readings: (Gebremedhin & Walther, 2020); Optional additional readings: (Rumelhart et al., 1986); (Baydin et al. 2017); (Maas et al. 2013); (He et al. 2015);
2024-10-18	Automatic differentiation in TensorFlow and in PyTorch. The multilayered perceptron in TensorFlow/Keras	ZLLS23 2.5, 5 Required additional readings: (Video lecture 2); (Video lecture 3 src);
2024-10-18	Explicit regularization by penalties. Effects of ridge (L2) and L1 regularizers. Weight decay. Bias-variance tradeoff and double descent. Bayesian interpretation and max-a-posteriori.	BB24 9.2, 4.3, 9.3.2, 2.6; GBC16 7.1, 7.2, 5.6, 7.8, 7. Optional: BB24 11 if you are unfamiliar with graphical models Optional additional readings: (Belkin et al. 2019);
2024-10-22	The Adam and the AdamW optimizers. Weight sharing. Early stopping. Dropout.	BB24 7.3, 9.3, 9.6; GBC16 7.8, 7.9, 8.5; Required additional readings: (Srivastava et al. 2014); (Loshchilov & Hutter 2018); Optional additional readings: (Baldi & Sadowski 2014); (Wager et al. 2013);
2024-10-25	More activation units (GELU, SiLU, Swish) and their relatioship to dropout. Weight initialization (LeCun, Glorot, He). Batch and layer normalization.	BB24 7.2.5, 7.4, 9.1.3; GBC16 7.4, 8.4, 8.7.1 Required additional readings: (Hendrycks & Gimpel 2020); (Ioffe & Szegedy 2015); (Ba et al. 2016); Optional additional readings: (Elfwing et al. 2018); (He et al. 2015); (Salimans et al. 2016); (Ioffe 2017);
2024-10-29	Convolutional networks for Nd signals and their inductive bias. Basic concepts and some variants. Translational equivariance. Stacking convolutional layers. Strides and pooling.	BB24 10.1, 10.2. GBC16 9.1-9.3, 9.5 Required additional readings: (Dumoulin & Visin 2016); Optional additional readings: (Ngiam et al. 2010); (Yu & Koltun 2015);
2024-11-05	Dilated and transposed convolutions. Bottlenecks (1x1 convolutions). Normalization for CNNs: batch, layer, instance, group. Gates. Mixtures of experts. Skip connections: Highway and residual networks. Sketch of DenseNet. Efficient net.	P24 10.2.6, 10.4, 11.2, 11.5.1, 11.5.2. GBC16 9.5, 9.8; ZLLS23 8.5-8.8 Required additional readings: (Wu & He 2018); Optional additional readings: (Yu & Koltun 2015); (Radford et al. 2015); (Jacobs et al. 1991); (Srivastava et al. 2018); (He et al. 2015); (Huang et al. 2017); (Tan & Le 2019);
2024-11-08	Semantic segmentation and U-nets. Fully convolutional structures. Class imbalance in segmentation problems and Dice loss. Sequence processing problems. Recurrent neural networks. Vanishing/exploding gradients when storing long-term information. Gates in RNNs: LSTM and GRU.	GBC16 10.1, 10.2; BB12.2.5; ZLLS23 9.4 Required additional readings: (Ronneberger et al. 2015); Optional additional readings: (Milletari et al. 2016); (Hochreiter et al. 2001); (Chung et al. 2014); (Hochreiter & Schmidhuber 1997);
2023-11-12	No class today. Also: office hours moved to Friday Nov 15, 14:15-16:15
2024-11-14	Dataloaders in PyTorch. Convolutional networks and DenseNet in PyTorch.	ZLLS23 7 Required additional readings: (Video lecture 4); (Video lecture 4 src); (Huang et al. 2017); Optional additional readings: (Walmsley et al. 2021); (Hui et al. 2022);
2024-11-15	Bidirectional RNNs. The general sequence-to-sequence learning problem. Encoder-decoder architectures. Decoding: Greedy; optimal decoding with Viterbi; beam search; sampling. Recurrent language models with attention. Embedding layers.	BB24 11.3, 12.1, 12.2; JM24 8.4.2, 8.7, 8.8, 13.2, 17.4 Optional additional readings: (Bahdanau et al. 2014); (Sutskever et al. 2014); (Graves et al. 2014);
2024-11-19	Introduction to transformers. Soft dictionaries. Self-attention layers. Complexity. Handling minibatches: Masking and batch matrix multiplication.	BB24 12.1.1-12.1.5,12.1.8; ZLLS23 11.1-11.3; JM24 9.1 Required additional readings: (Vaswani et al. 2017);
2024-11-22	Uso di einsum e einops. Multihead attention. Positional encoding.	BB24 12.1.7, 12.1.9, 12.3.1, 12.3.4 Required additional readings: (Rogozhnikov 2022); (Vaswani et al. 2017); Optional additional readings: (Dangel 2024);
2024-11-26	Vision Transformers. BERT. The pretraining/fine-tuning strategy. Transfer learning. Fine-tuning. Self-supervised learning. Pretext tasks.	Required additional readings: (Dosovitskiy et al. 2020); Optional additional readings: (Devlin et al. 2018); (Chopra et al. 2005); (Sohn 2016); (Van den Oord et al. 2019); (Doersch et al. 2016); (Gidaris et al. 2018); (He et al. 2022);
2024-11-29	The hyperparameter optimization problem. Elementary algorithm. An introduction to Gaussian processes.	B06 2.3, 3.3; RW06 1,2,4 Required additional readings: (Video lecture 5); Optional additional readings: (Franceschi et al. 2024); Online resources: (GPyTorch); (GPFlow);
2024-12-03	Triplet loss. Siamese networks. Contrastive learning. SimCLR. Model-based hyperparameter optimization.	Required additional readings: (Schroff et al. 2015); (Chen et al. 2020); (Snoek et al. 2012); Optional additional readings: (Chopra et al. 2005); (He et al. 2020); (Sohn 2016); (Yogatama et al. 2015); (Bergstra et al. 2013); (Jones et al. 1998); (Frazier 2018);
2024-12-06	Acquisition functions: probability of improvement, expected improvement. Multi-fidelity approaches to hyperparameter optimization. Successive halving. Hyperband. Brief mention of ASHA.	Required additional readings: (Bergstra et al. 2013); (Li et al. 2018); Optional additional readings: (Jamieson & Talwalkar 2016); (Li et al. 2020); Online resources: (GPyOpt); (Optuna); (Syne Tune);
2024-12-10	Overview of setting involving several distributions. Multi-task learning. Meta-learning. Out-of-distribution data and anomaly detection. Algorithms for domain adaptation. Reweighting examples. Mixup.	Required additional readings: (Ganin et al. 2016); (Zhang et al. 2018); Optional additional readings: (Ruder 2017); (Misra et al. 2016); (Ben-David et al. 2010); (Rozantsev et al. 2019); (Tzeng et al. 2017); (Li et al. 2017); (Finn et al. 2017); (Franceschi et al. 2018);

Note

Full text of linked papers is normally accessible when connecting from a UNIFI IP address. Use proxy-auth.unifi.it:8888 (with your credentials) if you are connecting from outside the campus network.