Supervisor

Suitable for

Abstract

Abstract

Discrete Flow Matching (DFM) is a recently introduced framework for non-autoregressive generation over discrete sequences such as text or code. In contrast to continuous flow models, which evolve samples within a continuous space, DFM defines a flow over discrete state spaces through probability paths that connect a simple source distribution to the data distribution. A key modelling choice is how this path is constructed, including the scheduling function, the coupling between source and target sequences, and the form of intermediate conditional distributions. Prior work has shown that these design decisions can affect perplexity, convergence behaviour and sampling stability, even when the underlying model remains unchanged.

 

This project will examine how different definitions of probability paths influence the performance of discrete flow models on small-scale datasets. The work will involve implementing DFM with several path choices and scheduling strategies, and comparing their effects on sampling quality, convergence and robustness. The goal is to understand how path design shapes the behaviour of discrete flows and to identify settings that lead to more reliable or efficient generation.

 

Pre-requisites:

Suitable for those who have taken a course in machine learning. Some familiarity with PyTorch would be beneficial.

 

References:

[1] Lipman, Yaron, et al. "Flow matching for generative modeling." International Conference on Learning Representations (ICLR), 2023. arXiv:2210.02747.

[2] Gat, Itai, et al. "Discrete flow matching." Advances in Neural Information Processing Systems 37 (NeurIPS), 2024.

[3] Austin, Jacob, et al. "Structured denoising diffusion models in discrete state-spaces." Advances in neural information processing systems 34 (NeurIPS), 2021.