RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación
Artículo científico (JournalArticle)
The Cambrian Explosion of Mixed-Precision Matrix Multiplication for Quantized Deep Learning Inference
[EN] Recent advances in deep learning (DL) have promoted to a shift from traditional 64-bit floating point (FP64) arithmetic for scientific computing toward reduced-precision formats-such as FP16, BF16, or even 8-bit integers-combined with mixed-precision arithmetic. This transition enhances computational throughput, reduces memory and bandwidth usage, and improves energy efficiency, offering significant advantages for resource-constrained edge devices. To support this shift, hardware architectures have evolved accordingly, now including adapted ISAs (Instruction Set Architectures) that expose mixed-precision vector units and matrix engines tailored for DL workloads. At the heart of many DL and scientific computing tasks is the general matrix-matrix multiplication (GEMM), a fundamental kernel historically optimized using fused multiply-add (FMA) vector instructions on SIMD (single instruction, multiple data) units. However, as hardware moves toward mixed-precision dot (or inner)-product-centric operations optimized for quantized inference, these legacy approaches are being phased out. In response to this, our paper revisits the conventional, high-performance implementation of GEMM and describes strategies for adapting it to mixed integer precision (MIP) arithmetic across modern ISAs, including x86_64, Arm, and RISC-V. Concretely, we illustrate novel micro-kernel designs and data layouts that better exploit today's specialized hardware and demonstrate significant performance gains from MIP arithmetic over floatingpoint implementations across three representative CPUs. These contributions highlight a new era of GEMM optimization-driven by the demands of DL inference on heterogeneous architectures, marking what we term as the "Cambrian period" for matrix multiplication., This work is supported by grants PID2021-126576NB-I00, PID2023-146569NB-C2 and PID2024-158311NB-I00, funded by MCIN/AEI/10.13039/5011000 11033, by "ERDF A way of making Europe", Programa PROMETEO2023 (CIPROM/2022/20) by Conselleria de Innovacio, Universidades, Ciencia y Sociedad Digital from the Generalitat Valenciana, and by Ayudaa Primeros Proyectosde Investigacion (PAID-06-24) , Vicerrectorado de Investigacion de la Universitat Politecnica de Valencia. A. Castello is supported by the CIAPOS/2023/431 grant, and Fondo Social Europeo Plus 2021-2027 from the Generalitat Valenciana.
Proyecto:
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2021-126576NB-I00
,
Agencia Estatal de Investigación / Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2023-146569NB-C21
,
Agencia Estatal de Investigación TED2021-129334B-I00
,
Generalitat Valenciana CIAPOS/2023/431
,
Universitat Politècnica de València PAID-06-24
,
Generalitat Valenciana CIPROM/2022/20
,
Agencia Estatal de Investigación PID2024-158311NB-I00