Uncertainty in Structured Prediction - exploring the limits of scale

Uncertainty estimation is important for ensuring safety and robustness of AI systems.  While most research in the area has focused on un-structured prediction tasks, limited work has investigated general uncertainty estimation approaches for structured prediction. Our work aims to investigate uncertainty estimation for autoregressive structured prediction tasks within a single unified and interpretable probabilistic ensemble-based framework.  We consider: uncertainty estimation for sequence data at the token-level and complete sequence-level; interpretations for, and applications of, various measures of uncertainty; and discuss both the theoretical and practical challenges associated with obtaining them. We also provide baselines for token-level and sequence-level error detection, and sequence-level out-of-domain input detection on the WMT’14 English-French and WMT’17 English-German translation and LibriSpeech speech recognition datasets. 

However, ensembles are computationally expensive to both construct and infer. Thus, we additionally explore methods of efficient ensemble generation and inference. Specifically, we describe a technique called 'Sequence Ensemble Distribution Distillation' - a distillation technique which enables a single model to fully emulate the original ensemble at low computational and memory cost.

LINK:https://teams.microsoft.com/l/meetup-join/19%3ameeting_MDU4MjQyN2QtMDQ1NC00YjdjLWFjYWYtNTE3ZTczZDdhZGI5%40thread.v2/0?context=%7b%22Tid%22%3a%22cc95de1b-97f5-4f93-b4ba-fe68b852cf91%22%2c%22Oid%22%3a%22260e1c37-d9de-4043-aaf6-80e297344503%22%7d