Pasar al contenido principal

Resultados relacionados

Resultados totales (Incluyendo duplicados): 21
Encontrada(s) 1 página(s)

UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Mind the memory gap: unveiling GPU bottlenecks in large-batch LLM inference

  • Garcia Recasens, Pol|||0009-0008-5535-0012
  • Agulló López, Ferran|||0000-0002-7276-2472
  • Zhu, Yue
  • Wang, Changyang
  • Lee, Eun Kyung
  • Tardieu, Olivier
  • Torres Tomás, Juan José
  • Berral García, Josep Lluís|||0000-0003-3037-3580
Large language models have been widely adopted across different tasks, but their auto-regressive generation nature often leads to inefficient resource utilization during inference. While batching is commonly used to increase throughput, per-formance gains plateau beyond a certain batch size, especially with smaller models, a phenomenon that existing literature typically explains as a shift to the compute-bound regime. In this paper, through an in-depth GPU-level analysis, we reveal that large-batch inference remains memory-bound, with most GPU compute capabilities underutilized due to DRAM bandwidth saturation as the primary bottleneck. To address this, we propose a Batching Configuration Advisor (BCA) that optimizes memory allocation, reducing GPU memory requirements with minimal impact on throughput. The freed memory and underutilized GPU compute capabilities can then be leveraged by concurrent workloads. Specifically, we use model replication to improve serving throughput and GPU utilization. Our findings challenge conventional assumptions about LLM inference, offering new in-sights and practical strategies for improving resource utilization, particularly for smaller language models., This work has been partially financed by grant agreement EU-HORIZON GA.101095717 and by the EU-HORIZON MSCA programme under grant agreement EU-HORIZON MSCA GA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478, by Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2021-126248OBI00, MCIN/AEI/10.13039/ 501100011033/ FEDER, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Dynamic in-node group-aware scheduling for multi-tenant machine learning services on Kubernetes

  • Liu, Peini|||0000-0003-0058-8732
  • Guitart Fernández, Jordi|||0000-0003-0751-3100
Machine Learning (ML) is becoming pervasive and integrated into different kinds of businesses. Hosting multi-tenant ML services on a unified platform requires efficient orchestration. Service orchestration considers two levels: an in-cluster scheduling to decide the node allocation, followed by an in-node scheduling to manage the resource distribution within the node. Previous works introduced multi-container deployments for ML services, demonstrating that partitioning the ML service and enabling CPU/Memory affinity for each container improves performance. These multi-container deployments have been enabled for Kubernetes at in-cluster level to allocate multiple groups of containers across nodes. However, when the containers are launched in the node, the in-node scheduler lacks awareness of the group information which challenges those containers for a fine-grained resource allocation, especially when multiple groups of containers share the same node. This paper presents an in-node group-aware scheduling mechanism for multiple ML services, where each service scheduling contains a group resource selection and container resource assignment. Moreover, we also provide a dynamic resource controller (DRC) to dynamically reallocate the resource allocation for containers using the mechanism, which monitors the group changes and acts with the real containers' system cgroups adaptation. Our results show that for deploying ML services with the same ML model and different ML models, DRC throughput outperforms other deployment scenarios by up to 258% and 319% respectively. In an experiment deploying a dynamic workload of multiple mixed ML services, DRC outperforms baseline NONE-Single by 242 %, 75 %, and 28 % for the average throughput of services with Mobilenet, Resnet50, and VGG 16, respectively, and also DRC results in a makespan 44% faster than NONE-Single and 4% faster than CM-Multi., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON GA.101092646, EU-HORIZON MSCA GA.101086248, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Data-driven optimization of GPU efficiency for distributed LLM-adapter serving

  • Agulló López, Ferran|||0000-0002-7276-2472
  • Oliveras Torra, Joan
  • Wang, Chen
  • Gutiérrez Torre, Alberto|||0000-0002-5548-3359
  • Tardieu, Olivier
  • Youssef, Alaa
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Berral García, Josep Lluís|||0000-0003-3037-3580
Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently. While prior work has largely focused on latency and throughput optimization, minimizing GPU resource requirements through near-peak utilization remains largely underexplored. This paper presents a data-driven pipeline that, for a given workload, computes an adapter placement that serves the workload with the minimum number of GPUs while avoiding request starvation and GPU memory errors. To that end, the approach identifies the maximum feasible throughput attainable on each GPU by leveraging accurate performance predictions learned from real serving behavior. The proposed pipeline integrates three components: (i) a Digital Twin (DT) tailored to LLM-adapter serving, (ii) a distilled machine learning (ML) model trained on DT-generated data, and (iii) a greedy placement algorithm that exploits ML-based performance estimates to maximize GPU efficiency. The DT emulates real system dynamics with high fidelity, achieving below 5% throughput estimation error while executing up to 90×faster than full LLM benchmarking across both predictable and unpredictable workloads. The learned ML models further accelerate performance estimation with marginal accuracy degradation, enabling scalable optimization. Experimental results demonstrate that the pipeline substantially improves GPU efficiency, reducing the number of GPUs required to sustain target workloads by 60% on average across the evaluated scenarios. Beyond GPU efficiency, the pipeline can be adapted to alternative objectives, such as latency minimization, highlighting its versatility for future large-scale LLM serving infrastructures., This work has been partially financed by the EU-HORIZON MSCA programme under grant agreement GA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478, by Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2024-160996OB-I00, MICIU/AEI/10.13039/501100011033/FEDER, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Data-Connector: an agent-based framework for autonomous ML-based smart management in cloud-edge continuum

  • Liu, Peini|||0000-0003-0058-8732
  • Oliveras Torra, Joan
  • Palacín Marfil, Marc
  • Guitart Fernández, Jordi|||0000-0003-0751-3100
  • Berral García, Josep Lluís|||0000-0003-3037-3580
  • Nou Castell, Ramon|||0000-0003-0464-1473
Machine Learning (ML) is becoming pervasive and integrated into different kinds of intelligent applications, and the collaborative Cloud-Edge continuum has been introduced as an emerging trend to support their adoption into use cases. However, managing these ML applications in the CloudEdge continuum is challenging due to the ML application's dynamic resource usage with different user loads and Cloud and Edge's dynamic resource availability. We envision machine learning methods that can be used for smart management in this dynamic environment, but how to deploy and utilize them for the adaptation scenario in Cloud-Edge continuum is unknown. This paper proposes an agent-based framework to enable autonomous smart management mechanisms that can be broadly enabled in diverse adaptation scenarios. The agent acts as a data-connector1 1https://github.com/bsc-scanflow/data-connector, connecting different sources of data, utilizing ML models for decision-making and triggering adaptations in Cloud-edge platforms. The case study shows the feasibility of our proposed data-connector for smart migration of an ML workload in the Cloud-edge continuum. The result shows that the smart migration-enabled Cloud-edge scenario has 11.9% ML application prediction time better than the Cloud scenario without migration. Moreover, with minimal customization, the data connector agent can be adapted for more use cases., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON GA.101092646, EU-HORIZON MSCA GA.101086248, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

A data-driven ML approach for maximizing performance in LLM-adapter serving

  • Agulló López, Ferran|||0000-0002-7276-2472
  • Oliveras Torra, Joan
  • Wang, Chen
  • Gutiérrez Torre, Alberto|||0000-0002-5548-3359
  • Tardieu, Olivier
  • Youssef, Alaa
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Berral García, Josep Lluís|||0000-0003-3037-3580
With the rapid adoption of Large Language Models (LLMs), LLM-adapters have become increasingly common, providing lightweight specialization of large-scale models. Serving hundreds or thousands of these adapters on a single GPU allows request aggregation, increasing throughput, but may also cause request starvation if GPU memory limits are exceeded. To address this issue, this study focuses on determining the joint configuration of concurrent and parallel adapters that maximizes GPU throughput without inducing starvation, given heterogeneous adapter and traffic properties. We propose a data-driven ML approach leveraging interpretable models to tackle this caching problem and introduce the first Digital Twin capable of reproducing an LLM-adapter serving system, enabling efficient training data generation. Experiments with the vLLM framework and LoRA adapters show that the Digital Twin reproduces throughput within 5.1% of real results, while the ML approach predicts optimal numbers of concurrent and parallel adapters with an error of at most 7.2% under heterogeneous, real-world workloads., This work has been partially financed by the EU-HORIZON MSCA programme under grant agreement EU-HORIZON MSCAGA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR)under grant agreement 2021-SGR-00478, by Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/ FEDER, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Artículo científico (JournalArticle)

Enhancing the output of time series forecasting algorithms for cloud resource provisioning

  • Agulló López, Ferran|||0000-0002-7276-2472
  • Gutiérrez Torre, Alberto|||0000-0002-5548-3359
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Berral García, Josep Lluís|||0000-0003-3037-3580
Forecasting the resource consumption of workloads is a frequent approach in the cloud provisioning field. Ideally, such predictions allow obtaining a more accurate scheduling and management of resources in a computing cluster. However, the current approaches fail to properly forecast the future consumption in areas where sudden increases of consumption are present, i.e., spikes. Even, commonly employed metrics lack the ability to properly evaluate sharp behaviours in the traces. This may generate resource starvation problems in the running workloads and decreases the Quality of Service (QoS) provided to external users. To address this issue, we propose two strategies that modify the outputs of forecasting algorithms without changing the algorithms’ internals. The new outputs considerably enhance the prediction of sudden increases, duplicating the F1 score metric in average for all tested algorithms. This improvement in the handling of spikes comes with an increased over-provision of resources. Nevertheless, the proposed strategies give the user an easy way to control this trade-off between predicting spikes and the amount of over-provision. The user can decide which is the right balance that better fits the requirements of its specific scenario. Furthermore, we propose a new evaluation methodology that better assesses the behaviour of forecasting algorithms in cloud traces, especially focused on the performance around increases of consumption, and we give insights on the reasons behind the predictions of the algorithms with the application of explainability techniques. The code repository of this work can be accessed through GitHub at this link https://github.com/FerranAgulloLopez/ResourceForecasting., This work has been supported by the EU’s Horizon research and innovation programme under grant agreements HORIZON GA.101092646 (CloudSkin), HORIZON MSCA GA.101086248 (CloudStars), and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreements PID-2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033, UE (DALEST), Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3 and the Generalitat de Catalunya (AGAUR), Spain 2021-SGR-00478 (CROMAI)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

CloudSkin: ML-based smart management for the cloud-edge continuum

  • Liu, Peini|||0000-0003-0058-8732
  • Nestorov, Anna Maria
  • Palacín Marfil, Marc
  • Nou Castell, Ramon|||0000-0003-0464-1473
  • Guitart Fernández, Jordi|||0000-0003-0751-3100
  • Berral García, Josep Lluís|||0000-0003-3037-3580
The CloudSkin project embraces ML-based autonomous management technologies, introducing the Learning Plane (LP), which provides services and methods to embed intelligence in Cloud-Edge Continuum management layers. By leveraging AI, the continuum can achieve user demands for resource, privacy, resilience, and energy efficiency requirements, in an adaptive way. As a first Proof-of-Concept, we envision a smart orchestrator to drive a 5G-powered automotive usecase, focusing on a mobility scenario where the LP optimises resources for AI data and applications, on a transversal Edge environment., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON GA.101092646, EU-HORIZON MSCA GA.101086248, EU-HORIZON GA.101092644, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

GreenWise: intelligent application migration for containerized machine learning services in the computing continuum

  • Liu, Peini|||0000-0003-0058-8732
  • Santos, José
  • De Turck, Filip
  • Guitart Fernández, Jordi|||0000-0003-0751-3100
Machine Learning (ML) services require both responsiveness and efficiency. In the dynamic Computing Continuum (CC), service migration has emerged as an important strategy to optimize service performance, improve power efficiency, or reduce operational costs. A basis enabler of the service migration in the CC is containerization and container orchestration, however, dynamically moving the service among Cloud and Edge servers based on several uncertainties is still challenging. In this paper, we present GreenWise, a framework for intelligent service migration for containerized ML services in the heterogeneous CC. GreenWise extends the monitoring agents to continuously monitor power consumption and performance metrics across diverse layers and sources in real-time, providing a holistic view of states. Leveraging Reinforcement Learning (RL), it enables Power-Performance aware strategies to achieve near-optimal online migration decisions under dynamic conditions. We perform GreenWise on top of a Kubernetes-based CC platform, implementing agents for intelligent migration for containerizedML services. Experimental results demonstrate that GreenWise achieves effective trade-offs between performance and power efficiency. Our proposed Power-Latency-Aware (MaskPPO) migration strategy outperforms Random and Round-Robin baselines 159.2% and 13.8% in a real cluster. These results highlight GreenWise’s potential for sustainable and intelligent ML service migration in the dynamic CC., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON CLOUDSKIN GA.101092646, EU-HORIZON SPIRIT GA.101070672, EU-HORIZON MSCA CLOUDSTARS GA.101086248; by the Spanish Ministry of Science (MICINN) PID2021-126248OB-I00, MCIN/AEI/10.13039/501100011033/FEDER, UE and PID2024-160996OB-I00; and by the Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478. José Santos is funded by the Research Foundation Flanders (FWO), grant numbers 1299323N and 1253226N., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

A reliability score for video-analytics edge-IoT devices in urban environments

  • Garcia Boris, Marçal
  • Oliveras Torra, Joan
  • Aguilera Luzón, David|||0009-0001-1302-9000
  • Liu, Peini|||0000-0003-0058-8732
  • Divan Koller, Mario José
  • Berral García, Josep Lluís|||0000-0003-3037-3580
Performing video-analytics in Edge and IoT devices towards urban mobility allows computing near-data, providing fast and autonomous management from streets and roads. Thanks to advances in artificial intelligence methods, real-time classification of camera-provided image feeds can be performed in low-power devices to be deployed next to urban signaling and city services. However, devices deployed “in the wild” are affected by environmental factors such as temperature, either ambient or retained by its container, humidity or voltage changes, coming from sensorized urban furniture. Deciding where to deploy video-analytics workload requires a minimal device performance reliability, assuring that load can be processed in-place, before moving it towards the Fog/Cloud or other nearby devices. Here we present a simplistic factorization of a reliability Score to be quickly computed in Edge nodes, towards orchestrating workload according to the node surrounding environment. The score takes into account ambient temperature and humidity, also received voltage. The scoring is evaluated performing video-analytics workloads on real devices under physical stress, observing the performance degradation related to the external and internal factors applied onto the device. From the study, we observe how the different factors affect the functioning mode and quality of service, aside of indicating an estimation of the reliability of the urban-placed devices., This work is financed by the INTEL UFunding #14780, the EU-HORIZON and MSCA programmes under GA.101092646 and GA.101086248, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

CloudSkin: AI-based learning plane for autonomic management in the cloud-edge continuum

  • Liu, Peini|||0000-0003-0058-8732
  • Oliveras Torra, Joan
  • Palacín Marfil, Marc
  • Nou Castell, Ramon|||0000-0003-0464-1473
  • Berral García, Josep Lluís|||0000-0003-3037-3580
  • Guitart Fernández, Jordi|||0000-0003-0751-3100
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., CloudSkin aims to create a cognitive cloud-edge continuum platform that optimally exploits heterogeneous resources. Using AI, the platform automatically adapts to system and application behavior, and enables secure and seamless service deployment. Barcelona Supercomputing Center has led the design and development of an AI-based “Learning Plane” to automatically and continuously manage services in the cloudedge continuum to adapt to the dynamic environment. In the project, we enabled basic system models and workload characterization including regressions and time-series models, different levels of smart policies including heuristics and reinforcement learning, and we developed a data-connector agent to leverage those mechanisms towards real-world use cases. In this paper, we report on the technical results and insights we have explored within the project, explain how they are integrated as a learning plane to support CloudSkin use cases, and finally outline new open research lines., This work has been supported by the EU Horizon research and innovation programme under grant agreements HORIZON GA.101092646 (CloudSkin), HORIZON GA.101296742 (OptimAIse), HORIZON MSCA GA.101086248 (CloudStars), by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI), and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2024-160996OB-I00, MICIU/AEI/10.13039/501100011033 (SCALEST), and by the Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478 (CROMAI)., Peer Reviewed
UPCommons. Portal del coneixement obert de la UPC
Publicación Contribución de congreso (ConferenceOutput)

Towards Pareto optimal throughput in small language model serving

  • Garcia Recasens, Pol|||0009-0008-5535-0012
  • Zhu, Yue
  • Wang, Chen
  • Lee, Eun Kyung
  • Tardieu, Olivier
  • Youssef, Alaa
  • Torres Viñals, Jordi|||0000-0003-1963-7418
  • Berral García, Josep Lluís|||0000-0003-3037-3580
Large language models (LLMs) have revolutionized the state-of-the-art of many different natural language processing tasks. Although serving LLMs is computationally and memory demanding, the rise of Small Language Models (SLMs) offers new opportunities for resource-constrained users, who now are able to serve small models with cutting-edge performance. In this paper, we present a set of experiments designed to benchmark SLM inference at performance and energy levels. Our analysis provides a new perspective in serving, highlighting that the small memory footprint of SLMs allows for reaching the Pareto-optimal throughput within the resource capacity of a single accelerator. In this regard, we present an initial set of findings demonstrating how model replication can effectively improve resource utilization for serving SLMs., This work has been partially financed by grant agreement EUHORIZON GA.101095717 and by the EU-HORIZON MSCA programme under grant agreement EU-HORIZON MSCA GA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021- SGR-00478, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2021-126248OB-I00, MCIN/AEI/ 10.13039/ 501100011033/ FEDER, UE., Peer Reviewed
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

GreenWise: intelligent application migration for containerized machine learning services in the computing continuum

  • Liu, Peini
  • Santos, José
  • De Turck, Filip
  • Guitart Fernández, Jordi
Machine Learning (ML) services require both responsiveness and efficiency. In the dynamic Computing Continuum (CC), service migration has emerged as an important strategy to optimize service performance, improve power efficiency, or reduce operational costs. A basis enabler of the service migration in the CC is containerization and container orchestration, however, dynamically moving the service among Cloud and Edge servers based on several uncertainties is still challenging. In this paper, we present GreenWise, a framework for intelligent service migration for containerized ML services in the heterogeneous CC. GreenWise extends the monitoring agents to continuously monitor power consumption and performance metrics across diverse layers and sources in real-time, providing a holistic view of states. Leveraging Reinforcement Learning (RL), it enables Power-Performance aware strategies to achieve near-optimal online migration decisions under dynamic conditions. We perform GreenWise on top of a Kubernetes-based CC platform, implementing agents for intelligent migration for containerizedML services. Experimental results demonstrate that GreenWise achieves effective trade-offs between performance and power efficiency. Our proposed Power-Latency-Aware (MaskPPO) migration strategy outperforms Random and Round-Robin baselines 159.2% and 13.8% in a real cluster. These results highlight GreenWise’s potential for sustainable and intelligent ML service migration in the dynamic CC., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON CLOUDSKIN GA.101092646, EU-HORIZON SPIRIT GA.101070672, EU-HORIZON MSCA CLOUDSTARS GA.101086248; by the Spanish Ministry of Science (MICINN) PID2021-126248OB-I00, MCIN/AEI/10.13039/501100011033/FEDER, UE and PID2024-160996OB-I00; and by the Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478. José Santos is funded by the Research Foundation Flanders (FWO), grant numbers 1299323N and 1253226N., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

A data-driven ML approach for maximizing performance in LLM-adapter serving

  • Agulló López, Ferran
  • Oliveras Torra, Joan
  • Wang, Chen
  • Gutiérrez Torre, Alberto
  • Tardieu, Olivier
  • Youssef, Alaa
  • Torres Viñals, Jordi
  • Berral García, Josep Lluís
With the rapid adoption of Large Language Models (LLMs), LLM-adapters have become increasingly common, providing lightweight specialization of large-scale models. Serving hundreds or thousands of these adapters on a single GPU allows request aggregation, increasing throughput, but may also cause request starvation if GPU memory limits are exceeded. To address this issue, this study focuses on determining the joint configuration of concurrent and parallel adapters that maximizes GPU throughput without inducing starvation, given heterogeneous adapter and traffic properties. We propose a data-driven ML approach leveraging interpretable models to tackle this caching problem and introduce the first Digital Twin capable of reproducing an LLM-adapter serving system, enabling efficient training data generation. Experiments with the vLLM framework and LoRA adapters show that the Digital Twin reproduces throughput within 5.1% of real results, while the ML approach predicts optimal numbers of concurrent and parallel adapters with an error of at most 7.2% under heterogeneous, real-world workloads., This work has been partially financed by the EU-HORIZON MSCA programme under grant agreement EU-HORIZON MSCAGA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR)under grant agreement 2021-SGR-00478, by Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/ FEDER, UE., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Data-Connector: an agent-based framework for autonomous ML-based smart management in cloud-edge continuum

  • Liu, Peini
  • Oliveras Torra, Joan
  • Palacín Marfil, Marc
  • Guitart Fernández, Jordi
  • Berral García, Josep Lluís
  • Nou Castell, Ramon
Machine Learning (ML) is becoming pervasive and integrated into different kinds of intelligent applications, and the collaborative Cloud-Edge continuum has been introduced as an emerging trend to support their adoption into use cases. However, managing these ML applications in the CloudEdge continuum is challenging due to the ML application's dynamic resource usage with different user loads and Cloud and Edge's dynamic resource availability. We envision machine learning methods that can be used for smart management in this dynamic environment, but how to deploy and utilize them for the adaptation scenario in Cloud-Edge continuum is unknown. This paper proposes an agent-based framework to enable autonomous smart management mechanisms that can be broadly enabled in diverse adaptation scenarios. The agent acts as a data-connector1 1https://github.com/bsc-scanflow/data-connector, connecting different sources of data, utilizing ML models for decision-making and triggering adaptations in Cloud-edge platforms. The case study shows the feasibility of our proposed data-connector for smart migration of an ML workload in the Cloud-edge continuum. The result shows that the smart migration-enabled Cloud-edge scenario has 11.9% ML application prediction time better than the Cloud scenario without migration. Moreover, with minimal customization, the data connector agent can be adapted for more use cases., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON GA.101092646, EU-HORIZON MSCA GA.101086248, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

CloudSkin: ML-based smart management for the cloud-edge continuum

  • Liu, Peini
  • Nestorov, Anna Maria
  • Palacín Marfil, Marc
  • Nou Castell, Ramon
  • Guitart Fernández, Jordi
  • Berral García, Josep Lluís
The CloudSkin project embraces ML-based autonomous management technologies, introducing the Learning Plane (LP), which provides services and methods to embed intelligence in Cloud-Edge Continuum management layers. By leveraging AI, the continuum can achieve user demands for resource, privacy, resilience, and energy efficiency requirements, in an adaptive way. As a first Proof-of-Concept, we envision a smart orchestrator to drive a 5G-powered automotive usecase, focusing on a mobility scenario where the LP optimises resources for AI data and applications, on a transversal Edge environment., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON GA.101092646, EU-HORIZON MSCA GA.101086248, EU-HORIZON GA.101092644, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

A reliability score for video-analytics edge-IoT devices in urban environments

  • Garcia Boris, Marçal
  • Oliveras Torra, Joan
  • Aguilera Luzón, David
  • Liu, Peini
  • Divan Koller, Mario José
  • Berral García, Josep Lluís
Performing video-analytics in Edge and IoT devices towards urban mobility allows computing near-data, providing fast and autonomous management from streets and roads. Thanks to advances in artificial intelligence methods, real-time classification of camera-provided image feeds can be performed in low-power devices to be deployed next to urban signaling and city services. However, devices deployed “in the wild” are affected by environmental factors such as temperature, either ambient or retained by its container, humidity or voltage changes, coming from sensorized urban furniture. Deciding where to deploy video-analytics workload requires a minimal device performance reliability, assuring that load can be processed in-place, before moving it towards the Fog/Cloud or other nearby devices. Here we present a simplistic factorization of a reliability Score to be quickly computed in Edge nodes, towards orchestrating workload according to the node surrounding environment. The score takes into account ambient temperature and humidity, also received voltage. The scoring is evaluated performing video-analytics workloads on real devices under physical stress, observing the performance degradation related to the external and internal factors applied onto the device. From the study, we observe how the different factors affect the functioning mode and quality of service, aside of indicating an estimation of the reliability of the urban-placed devices., This work is financed by the INTEL UFunding #14780, the EU-HORIZON and MSCA programmes under GA.101092646 and GA.101086248, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Data-driven optimization of GPU efficiency for distributed LLM-adapter serving

  • Agulló López, Ferran
  • Oliveras Torra, Joan
  • Wang, Chen
  • Gutiérrez Torre, Alberto
  • Tardieu, Olivier
  • Youssef, Alaa
  • Torres Viñals, Jordi
  • Berral García, Josep Lluís
Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently. While prior work has largely focused on latency and throughput optimization, minimizing GPU resource requirements through near-peak utilization remains largely underexplored. This paper presents a data-driven pipeline that, for a given workload, computes an adapter placement that serves the workload with the minimum number of GPUs while avoiding request starvation and GPU memory errors. To that end, the approach identifies the maximum feasible throughput attainable on each GPU by leveraging accurate performance predictions learned from real serving behavior. The proposed pipeline integrates three components: (i) a Digital Twin (DT) tailored to LLM-adapter serving, (ii) a distilled machine learning (ML) model trained on DT-generated data, and (iii) a greedy placement algorithm that exploits ML-based performance estimates to maximize GPU efficiency. The DT emulates real system dynamics with high fidelity, achieving below 5% throughput estimation error while executing up to 90×faster than full LLM benchmarking across both predictable and unpredictable workloads. The learned ML models further accelerate performance estimation with marginal accuracy degradation, enabling scalable optimization. Experimental results demonstrate that the pipeline substantially improves GPU efficiency, reducing the number of GPUs required to sustain target workloads by 60% on average across the evaluated scenarios. Beyond GPU efficiency, the pipeline can be adapted to alternative objectives, such as latency minimization, highlighting its versatility for future large-scale LLM serving infrastructures., This work has been partially financed by the EU-HORIZON MSCA programme under grant agreement GA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478, by Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2024-160996OB-I00, MICIU/AEI/10.13039/501100011033/FEDER, UE., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Towards Pareto optimal throughput in small language model serving

  • Garcia Recasens, Pol
  • Zhu, Yue
  • Wang, Chen
  • Lee, Eun Kyung
  • Tardieu, Olivier
  • Youssef, Alaa
  • Torres Viñals, Jordi
  • Berral García, Josep Lluís
Large language models (LLMs) have revolutionized the state-of-the-art of many different natural language processing tasks. Although serving LLMs is computationally and memory demanding, the rise of Small Language Models (SLMs) offers new opportunities for resource-constrained users, who now are able to serve small models with cutting-edge performance. In this paper, we present a set of experiments designed to benchmark SLM inference at performance and energy levels. Our analysis provides a new perspective in serving, highlighting that the small memory footprint of SLMs allows for reaching the Pareto-optimal throughput within the resource capacity of a single accelerator. In this regard, we present an initial set of findings demonstrating how model replication can effectively improve resource utilization for serving SLMs., This work has been partially financed by grant agreement EUHORIZON GA.101095717 and by the EU-HORIZON MSCA programme under grant agreement EU-HORIZON MSCA GA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021- SGR-00478, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2021-126248OB-I00, MCIN/AEI/ 10.13039/ 501100011033/ FEDER, UE., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Dynamic in-node group-aware scheduling for multi-tenant machine learning services on Kubernetes

  • Liu, Peini
  • Guitart Fernández, Jordi
Machine Learning (ML) is becoming pervasive and integrated into different kinds of businesses. Hosting multi-tenant ML services on a unified platform requires efficient orchestration. Service orchestration considers two levels: an in-cluster scheduling to decide the node allocation, followed by an in-node scheduling to manage the resource distribution within the node. Previous works introduced multi-container deployments for ML services, demonstrating that partitioning the ML service and enabling CPU/Memory affinity for each container improves performance. These multi-container deployments have been enabled for Kubernetes at in-cluster level to allocate multiple groups of containers across nodes. However, when the containers are launched in the node, the in-node scheduler lacks awareness of the group information which challenges those containers for a fine-grained resource allocation, especially when multiple groups of containers share the same node. This paper presents an in-node group-aware scheduling mechanism for multiple ML services, where each service scheduling contains a group resource selection and container resource assignment. Moreover, we also provide a dynamic resource controller (DRC) to dynamically reallocate the resource allocation for containers using the mechanism, which monitors the group changes and acts with the real containers' system cgroups adaptation. Our results show that for deploying ML services with the same ML model and different ML models, DRC throughput outperforms other deployment scenarios by up to 258% and 319% respectively. In an experiment deploying a dynamic workload of multiple mixed ML services, DRC outperforms baseline NONE-Single by 242 %, 75 %, and 28 % for the average throughput of services with Mobilenet, Resnet50, and VGG 16, respectively, and also DRC results in a makespan 44% faster than NONE-Single and 4% faster than CM-Multi., This work is financed by the EU-HORIZON programme under grant agreements EU-HORIZON GA.101092646, EU-HORIZON MSCA GA.101086248, by Generalitat de Catalunya (AGAUR) GA.2021-SGR-00478, and the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) PID2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033/FEDER, UE., Peer Reviewed, Postprint (author's final draft)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Artículo científico (JournalArticle)

Enhancing the output of time series forecasting algorithms for cloud resource provisioning

  • Agulló López, Ferran
  • Gutiérrez Torre, Alberto
  • Torres Viñals, Jordi
  • Berral García, Josep Lluís
Forecasting the resource consumption of workloads is a frequent approach in the cloud provisioning field. Ideally, such predictions allow obtaining a more accurate scheduling and management of resources in a computing cluster. However, the current approaches fail to properly forecast the future consumption in areas where sudden increases of consumption are present, i.e., spikes. Even, commonly employed metrics lack the ability to properly evaluate sharp behaviours in the traces. This may generate resource starvation problems in the running workloads and decreases the Quality of Service (QoS) provided to external users. To address this issue, we propose two strategies that modify the outputs of forecasting algorithms without changing the algorithms’ internals. The new outputs considerably enhance the prediction of sudden increases, duplicating the F1 score metric in average for all tested algorithms. This improvement in the handling of spikes comes with an increased over-provision of resources. Nevertheless, the proposed strategies give the user an easy way to control this trade-off between predicting spikes and the amount of over-provision. The user can decide which is the right balance that better fits the requirements of its specific scenario. Furthermore, we propose a new evaluation methodology that better assesses the behaviour of forecasting algorithms in cloud traces, especially focused on the performance around increases of consumption, and we give insights on the reasons behind the predictions of the algorithms with the application of explainability techniques. The code repository of this work can be accessed through GitHub at this link https://github.com/FerranAgulloLopez/ResourceForecasting., This work has been supported by the EU’s Horizon research and innovation programme under grant agreements HORIZON GA.101092646 (CloudSkin), HORIZON MSCA GA.101086248 (CloudStars), and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreements PID-2021-126248OB-I00, MCIN/AEI/10.13039/ 501100011033, UE (DALEST), Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3 and the Generalitat de Catalunya (AGAUR), Spain 2021-SGR-00478 (CROMAI)., Peer Reviewed, Postprint (published version)
Recercat. Dipósit de la Recerca de Catalunya
Publicación Contribución de congreso (ConferenceOutput)

Mind the memory gap: unveiling GPU bottlenecks in large-batch LLM inference

  • Garcia Recasens, Pol
  • Agulló López, Ferran
  • Zhu, Yue
  • Wang, Changyang
  • Lee, Eun Kyung
  • Tardieu, Olivier
  • Torres Tomás, Juan José
  • Berral García, Josep Lluís
Large language models have been widely adopted across different tasks, but their auto-regressive generation nature often leads to inefficient resource utilization during inference. While batching is commonly used to increase throughput, per-formance gains plateau beyond a certain batch size, especially with smaller models, a phenomenon that existing literature typically explains as a shift to the compute-bound regime. In this paper, through an in-depth GPU-level analysis, we reveal that large-batch inference remains memory-bound, with most GPU compute capabilities underutilized due to DRAM bandwidth saturation as the primary bottleneck. To address this, we propose a Batching Configuration Advisor (BCA) that optimizes memory allocation, reducing GPU memory requirements with minimal impact on throughput. The freed memory and underutilized GPU compute capabilities can then be leveraged by concurrent workloads. Specifically, we use model replication to improve serving throughput and GPU utilization. Our findings challenge conventional assumptions about LLM inference, offering new in-sights and practical strategies for improving resource utilization, particularly for smaller language models., This work has been partially financed by grant agreement EU-HORIZON GA.101095717 and by the EU-HORIZON MSCA programme under grant agreement EU-HORIZON MSCA GA.101086248. Also, it has been partially financed by Generalitat de Catalunya (AGAUR) under grant agreement 2021-SGR-00478, by Severo Ochoa Center of Excellence CEX-2021-001148-S-20-3, and by the Spanish Ministry of Science (MICINN), the Research State Agency (AEI) and European Regional Development Funds (ERDF/FEDER) under grant agreement PID2021-126248OBI00, MCIN/AEI/10.13039/ 501100011033/ FEDER, UE., Peer Reviewed, Postprint (published version)