Dottorato in MATEMATICA

Abstract: The idea is to give the basic ingredients to understand what Deep Learning theory is, and of what are the underlying mathematical/theoretical structures. Hopefully at the end of the course, the mathematics students that follow it will be able to read Deep Learning research papers without being lost, and will have the basic tools to start working on some important maths-rich topics. Lectures 7-12 below, each will cover an important direction of research for which some first mathematical steps have been done, but which has many subtopics and extensions left open. Particular emphasis is given to the view that the geometry of datapoints and of learning algorithms has important practical consequences, some of which have started to emerge in the last 3-4 years.

The plan is to spend one lecture on each topic. The below plan may change as the course progresses:

Part I: Introduction to Deep Learning:

1. Introduction and a brief history of Neural Networks. Overview of the course.

2. Stochastic Gradient Descent, Backpropagation, Convergence improvement methods

3. Some very common Neural Network architectures and their motivations

4. Neuromorphic Neural Networks

Part II: Staples of classical Deep Learning theory

5. Regularization, Generalization: some mathematical interpretations

6. Expressivity, PAC learning, VC dimension and comparison to real life learning for DNNs

7. Introduction to Information Theory, and the Information Bottleneck Principle

Part III: Selected topics of research

8. Network pruning: the "Lottery ticket hypothesis", and directions for theoretical justifications

9. Equivariant Neural Networks

10. Curvature and its role in Generative Adversarial Networks

11. Hyperbolic Neural Networks

12. Persistence diagrams and Topological Data Analysis (guest lecture by Sara Scaramuccia)

Useful bibliography for the first 6 lectures (precise chapter references and other references will appear in the lecture slides):

• Deep Learning Book -- https://www.deeplearningbook.org/ (useful for lectures 1, 2, 5)

• Some famous Deep Learning architectures -- https://towardsdatascience.com/neural-network-architectures-156e5bad51ba (useful for lecture 3)

• [missing: survey paper on Neuromorphic Neural Networks] (useful for lecture 4)

• Foundations of Machine Learning -- https://cs.nyu.edu/~mohri/mlbook/ (useful for lecture 6)