Classical Machine Learning

Machine learning is an increasingly important aspect of software engineering as ML- based products and systems are entering in widespread use across industries and sectors. Our existing machine learning offering in PRO is DNN, a course that has witnessed among the highest enrolment demands of any course in recent years. However, the DNN course focuses exclusively on neural network methods, which means that students gain an in-depth, but somewhat myopic, view of machine learning.

This course seeks to fill in a gap that currently exists in PRO, towards three fundamental goals. One is to ease students new to machine learning to a curated selection of simple methods that both represent a diversity of fundamental approaches which are in common use today. The second is to get students hands-on experience with the practice of machine learning, through popular libraries and frameworks used today. The use of simple classifiers will allow students to fully explore these learning tools and frameworks with real datasets, to train, test, interrogate and compare models and their relative strengths and weaknesses. The final goal is to cover fundamental "big ideas" which apply regardless of method. This will include abstract concepts such as Ockham's Razor, the curse of dimensionality, the importance of feature representation, and No Free Lunch.

Establishing a broad introductory understanding of ML techniques is important for several reasons. Fundamental principles and techniques are important regardless of method applied, and understanding how techniques differ (and relate to the problems they are trying to solve) can give students invaluable intuition about the nature of their learning problems. Moreover, there are many cases in which classical ML techniques are still preferable to DNN- based methods. In some cases, classical ML techniques still either outperform, or perform on par with DNNs. The more common reason classical methods are often seen as preferable is that they are easier to interpret and validate, which is critical for many kinds of important applications, including safety-critical systems and those supporting high-stakes decisions.

Course dates

Objectives

As with the DNN course, the objective of the course is to give a taste of the theory and approach of each in a way that is suited to a general audience (i.e. without requiring significant mathematical background, including linear algebra), and to spend a significant amount of time working practical ML experiments and examples with these methods applied to real datasets. This hands-on experience will provide students with plenty of opportunity to experience the practical challenges of applying ML in practice, including incorrect or missing data, the need to think deeply about representations and transformation for creating features, and significant challenges around overfitting and generalisation.

Due to its importance to ML at present, we will do all practicals in Jupyter Notebook, with Python 3, scikit-learn, pandas and associated tooling.

At the conclusion of this course students should understand:

• The principles and approaches used within classical machine learning.
• The main variants of classical machine learning e.g. classification and regression/supervised and unsupervised approaches.
• The key concepts, issues and practices when modeling with these techniques; as well as have hands-on experience in using frameworks for this purpose.
• An understanding of key similarities and differences between deterministic and probabilistic approaches.
• How classical machine learning fits within the context of other machine learning approaches, and what tasks it is considered to be suited and not well suited to perform.

Contents

Below we briefly list the core topics that will be covered in the class, first describing the concepts, the specific learning methods, and what will be covered in practicals.

Fundamental concepts:

Formulations of the problem of learning: as interpolation/estimation, as search; as hypothesis testing, as concept generalisation.
Supervised, Unsupervised, Semi-supervised, Reinforcement Learning
Classification, Multi-Class Classification and Regression
Model performance: Accuracy, Generalisation, and Overfitting
Training vs Test Performance, Loss and reward functions
Features: Selection and representation
Ockham's Razor, No Free Lunch and their implications

Learning methods:

Linear and polynomial regression, logistic regression
k-Nearest Neighbours
Decision Trees
k-Means and Hierarchical Clustering
Dimensionality reduction with Principal Component Analysis
Basics of Probability and Bayes' Theorem
Bayesian Networks
The Naive-Bayes Classifier
Markov chains
Hidden Markov Models
Support Vector Machines
Genetic Algorithms

Practicals:

Introduction to Jupyter Notebook, and Scitkit learn:
        Core data structures and methods
        Data loading, processing, feature extraction
        Model training, testing, cross-validation, inspection
Logistic and Linear Regression Lab
        Hands on with over- and under-fitting
Decision Trees and kNN Lab
Clustering Lab
Bayesian Learning Lab
        Parameter learning in Bayes Nets
        Structure learning for Bayes Nets
        Comparing Bayes factorisation to complete joint probability
        Text Classification with Naive-Bayes
        Time series classification with HMMs
Support Vector Machines Lab
        Text classification (Detecting Spam Email) with SVMs
        Vision (Face detection) with SVMs
        Effects of Kernels and C
Optional: Genetic Algorithms Lab

Assignment

The assignment will test students’ theoretical and practical understanding of the topics covered in the course, through a mixture of dataset exploration and analysis, as well as explanations of different machine learning concepts.

Teaching methods

The main teaching methods will be lectures (up to 3 hours daily, broken into two 1.5 hour lectures each) presented using highly visual slides and a few whiteboard explanations. Hands-on practicals will make up the rest of the work; it will consist of a series of exercises applying methods covered using real datasets, Jupyter Notebook and Python, with supporting libraries. To make it easy for students to directly compare learning methods and approaches, practicals will be designed to build one upon the next on (re-using the same data sets of earlier practicals).

Requirements

The course is intended to be of an introductory nature, and accessible to all PRO students, regardless of mathematical background or previous experience. In order to keep it such, we will not be doing rigorous mathematical derivations of classifier properties, proofs of convergence or delve into detail in training algorithms; students enroling should not expect these details.