PyTorch 2.13 brings FlexAttention to MPS

PyTorch 2.13 shipped on July 8, assembled from 3,328 commits by 526 contributors. The most visible change for Apple Silicon users is FlexAttention now working on MPS: hand-written Metal kernels for sparse prefill and decode paths achieve up to roughly 12x the throughput of standard scaled dot-product attention on sparse patterns, with the gap widening as sparsity increases.

Three other additions carry real weight for training workloads. CuTeDSL is a new Python-native domain-specific language built on NVIDIA’s CuTe (CUDA Templates) that gives Inductor an alternative to Triton for GEMM and RMSNorm operations; it compiles faster by eliminating Python GIL bottlenecks and produces higher-quality matrix-multiply code. The fused nn.LinearCrossEntropyLoss cuts peak memory by up to 4x on large-vocabulary softmax by chunking the vocabulary dimension and never materializing the full logits matrix, making it a drop-in replacement for the separate linear-plus-cross-entropy pattern. And a new deterministic CUDA backward path replaces atomic operations with a pre-computed write order, giving bit-for-bit reproducible gradients at under 1% runtime overhead at typical sequence lengths; it activates via torch.use_deterministic_algorithms(True).

For teams running fine-tuning jobs or coding agents remotely, the memory savings and reproducibility changes are the most immediately practical. The Hugging Face Transformers 5.13 update covers the ecosystem tooling that goes alongside this release.