DISTAL: The Distributed Tensor Algebra Compiler
- 16:00 19th November 2021 ( week 6, Michaelmas Term 2021 )Zoom
We introduce DISTAL, a compiler for dense tensor algebra that targets modern distributed and heterogenous systems. DISTAL allows users to independently describe how tensors and computation map onto the target machine through the tensors’ formats and a scheduling language. The combination of choices for data and computation distribution creates a design space that includes algorithms from the past (Cannon’s algorithm) and present (COSMA). DISTAL compiles a tensor algebra domain specific language to a distributed task-based runtime system and supports nodes with multi-core CPUs and multiple GPUs. Code generated by DISTAL is competitive with optimized codes for matrix multiply on 256 nodes of the Lassen supercomputer and outperforms existing systems by between 1.8x to 3.7x (with a 45.7x outlier) on higher order tensor operations.