# Probabilistic ML course

## Table of contents

## Class Timings

Thursday, 12 PM to 1:30 PM, 7-102 Friday, 2 PM to 3:30 PM, 7-105

## Google Classroom Link

u7jjtu2

## Prerequisites

- Mathematics for ML: Recommended reading is MML book
- A prior ML course: ES654 course at IIT Gandhinagar or equivalent
- Good programming skills in Python. Familiarity with: numpy, Pandas, matplotlib, sklearn. Recommended reading is Python Data Science Handbook

## Course contents:

- Probability refresher: Probability theory, discrete distributions, continuous distributions, joint probability distributions, sampling from different distributions (e.g. using Box-Muller transform), uncertainty modelling, information theoretic concepts: (KL-divergence, entropy)
- Bayesian concept learning: Likelihood, Prior, Posterior, Maximum Likelihood Estimate (MLE), Maximum A-Posteriori estimation (MAP), Full Bayesian Estimation with Conjugate Priors (Beta-Bernoulli model for the coin toss example, Normal-Normal model for estimating parameters of Normal distribution)
- Bayesian supervised methods:
- Regression: Bayesian linear regression, Robust linear regression via alternative likelihood (e.g. Laplace, Student-T)
- Classification: Bayesian Logistic regression, Bayesian Naive Bayes

- Latent Variable modelling: Gaussian Mixture Model, Probabilistic principal component analysis (PPCA)
- Approximate Inference:
- Sampling based strategies: Rejection sampling, Importance sampling, Markov Chain Monte Carlo (MCMC), Metropolis Hastings, Gibbs sampling, No U-Turn sampler (NUTS)
- Variational inference: Mean field approach, Evidence Lower Bound (ELBO), Reparameterization trick, Stochastic Variational Inference, Automatic Differentiation Variational Inference (ADVI)

- Gaussian Processes (GP): Multivariate Normal distribution and its properties, Kernels, GP regression, GP classification, Approximate Inference and Sparse GPs
- Bayesian Optimization (BO) and Active Learning (AL): AL - Query by committee, Uncertainty sampling, Expected model change, BO - Acquisition functions, GP based BO, Random Forests based BO
- Probabilistic Deep Learning: MC Dropout, Deep Ensembles, Bayesian neural networks (BNNs), Deep GPs

## Textbooks

- Kevin Murphy. Machine Learning, A Probabilistic Perspective. The MIT Press, 2012.
- Kevin Murphy. Probabilistic Machine Learning: An Introduction. The MIT Press, 2022
- Kevin Murphy. Probabilistic Machine Learning: Advanced Topics. The MIT Press, 2023.
- Chris Bishop. Pattern Recognition and Machine Learning.
- Allen Downey. Think Bayes: Bayesian Statistics in Python. Green Tea Apress, 2012
- David Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012
- Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006
- Richard McLearth. Statistical Rethinking. CRC Press, 2020.

## Other similar courses

- PML course from Piyush Rai (IITK)

## Grading Policy

- Assignments: 20%
- Quizzes: 20%
- Quiz syllabus will be everything from previous quiz to present day

- Projects: 60%

## Useful YT channels/Playlists

- https://www.youtube.com/user/mathematicalmonk
- https://www.youtube.com/c/MutualInformation0
- https://www.youtube.com/c/ritvikmath
- https://www.youtube.com/c/MachineLearningSimulation
- https://www.youtube.com/watch?v=UbaVGD4Lfis&list=PL05umP7R6ij1tHaOFY96m5uX3J21a6yNd
- https://www.youtube.com/watch?v=BYUykHScxj8&list=PLDcUM9US4XdMROZ57-OIRtIK0aOynbgZN