# Bayesian Statistical Probabilistic Programming:  2021-2022

 Degrees Term Michaelmas Term 2021  (16 lectures)

## Overview

The lectures for this course will be pre-recorded.

They will be available on Panopto (click Recorded Lectures>2021-22>Bayesian Statistical Probabilistic Programming).

They will be supported by discussion on Tuesdays from 12pm-1pm in Lecture Theatre B

Statistical probabilistic programming (SPP) is a general framework for expressing probabilistic models as computer programs. SPP systems are equipped with implementations of genericinference algorithms for answering queries about these models, such as posterior inference and marginalisation. By providing these algorithms, SPP systems enable data scientists to focuson the design of good models, utilising their domain knowledge. The task of constructing efficient generic inference engines can be left to researchers with expertise in statistical machine learning and programming languages.

The full potential of SPP will come from automating the inference of latent (unobserved) variables in the model, conditioned on the observed data. Because of their generality, SPP poses interesting and challenging research problems for programming languages and the statistical foundations of machine learning.

## Learning outcomes

1. To construct probabilistic models via an expressive SPP language. We use the SPP language Anglican, but emphasise general principles that are applicable to a wide range of existing SPP languages.
2. To understand standard inference algorithms and their implementations in SPP languages; and to apply them to perform inference in a SPP system.
3. To understand the semantics of SPP programs, with a view to reasoning about their correctness.

## Prerequisites

- Prelims Functional Programming, Continuous Maths, Linear Algebra, and Probability
- Part B Machine Learning

Though not strict prerequisites, the following courses are highly desirable:
- Principle of Programming Languages
- Part A Probability (MCS). The following topics covered in the Part A course will be assumed: probability spaces and random variables, conditional probability, probability distributions, expectation and variance, convergence and limit theorems, joint distribution and conditional density, basics of Markov chains.

## Synopsis

• Basics of Anglican; examples of Bayesian inference problems.
• Review of (discrete) probability theory.
• Posterior inference; likelihood-weighted importance sampling and importance sampling. Linear regression.
• Generative modelling with Anglican: 2-D physics, and program induction.
• Markov Chain Monte Carlo algorithms.
• Implementing inference algorithms for SPP programs – an operational semantic account.
• Variational inference.
• Sequential Monte Carlo and amortised inference.
• Basic measure theory and programs as measures.
• Denotational semantics of SPP programs.

Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006. ISBN9788578110796. doi: 10.1017/CBO9781107415324.004.

Cameron Davidson-Pilon. Bayesian methods for Hackers. Addison-Wesley, 2015.

Zoubin Ghahramani. Probabilistic machine learning and artificial intelligence. Nature, 521(7553):452–459, 2015. ISSN 0028-0836. doi: 10.1038/nature14541. URL http://www.nature.com/doifinder/10.1038/nature14541.

Noah D Goodman and Andreas Stuhlmüller. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org, 2014.

Kevin P Murphy. Machine Learning: A Probabilistic Perspective. Adaptive Computation andMachine Learning Series. The MIT Press, Cambridge, Mass, 2012. ISBN 9780262018029.

Tom Rainforth. Automating Inference, Learning, and Design using Probabilistic Programming. PhD thesis, University of Oxford, 2017.

Stuart Russell. Unifying logic and probability. Communications of the ACM, 58(7), 2015. doi: 10.1145/2699411.

Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. An introduction to probabilistic programming. CoRR, abs/1809.10756, 2018. URL http://arxiv.org/abs/1809.10756.

Frank Wood et al. Anglican – A Probabilistic Programming System. Reference webpage, 2014–2018. URL https://probprog.github.io/anglican/index.html.