Pasar al contenido principal

INTELIGENCIA SOSTENIBLE EN EL BORDE-UJI

PID2023-146569NB-C22

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para Impulsar la Investigación Científico-Técnica y su Transferencia
Subprograma Subprograma Estatal de Generación de Conocimiento
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2023
Plan Estatal 2021-2023 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSITAT JAUME I DE CASTELLO

Resultados relacionados

Resultados totales (Incluyendo duplicados): 1
Encontrada(s) 1 página(s)

RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia
Publicación Artículo científico (JournalArticle)

Characterization of quantized inference with transformer encoders on low power CPUs

  • Martínez, Héctor
  • Catalán, Sandra
  • Castelló, Adrián|||0000-0002-8576-8451
  • Quintana-Ortí, Enrique S.|||0000-0002-5454-165X
[EN] Transformers, particularly large language models (LLMs), are revolutionizing applications in natural language processing and computer vision but at a high cost in memory, energy, and computational resources. Quantization has emerged as an effective compression method to alleviate these demands, reducing the bitwidth of model data and arithmetic precision to enable efficient inference on resource-constrained devices. This paper focuses on optimizing inference with transformer encoders on low-power general-purpose CPUs, as those often found in edge devices. Our key contributions include exposing the critical role of linear layers within transformer encoders on CPUs with a limited number of cores; developing mixed integer precision matrix multiplication on ARM and RISC-V CPUs; and evaluating performance impact and energy savings of quantized inference. In summary, this work highlights the advantages of applying quantization to transformer encoders on current single-core and multi-core low power CPUs, offering insights for efficient LLM deployment on edge platforms., The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: PID2023-146569NB-C2 and PID2020-113656RB-C22 of MCIN/AEI/10.13039/501100011033 and ERDF/UE, POSTDOC_21_00025 of Junta de Andalucia, RYC2021-033973-I of MCIN/AEI/10.13039/501100011033 and the "NextGenerationEU"/PRTR, UJI-2023-04 of Universitat Jaume I, and CIAPOS/2023/431 of Fondo Social Europeo Plus 2021-2027 from the Generalitat Valenciana.