DISTAL: The Distributed Tensor Algebra Compiler

We introduce DISTAL, a compiler for dense tensor algebra that targets modern distributed and heterogenous systems. DISTAL allows users to independently describe how tensors and computation map onto the target machine through the tensors’ formats and a scheduling language. The combination of choices for data and computation distribution creates a design space that includes algorithms from the past (Cannon’s algorithm) and present (COSMA). DISTAL compiles a tensor algebra domain specific language to a distributed task-based runtime system and supports nodes with multi-core CPUs and multiple GPUs. Code generated by DISTAL is competitive with optimized codes for matrix multiply on 256 nodes of the Lassen supercomputer and outperforms existing systems by between 1.8x to 3.7x (with a 45.7x outlier) on higher order tensor operations.