Pasar al contenido principal

CLUES TO GALAXY EVOLUTION IN EXTREME ENVIRONMENTS WITH SKA PRECURSORS AND REGIONAL CENTRES USING OPEN SCIENCE TOOLS

PID2024-155817OB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal para la Investigación y el Desarrollo Experimental
Subprograma Subprograma Estatal de Generación de Conocimiento Científico-Técnico y Desarrollo Experimental
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2024
Plan Estatal 2024-2027 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICAS (CSIC)

Resultados relacionados

Resultados totales (Incluyendo duplicados): 2
Encontrada(s) 1 página(s)

DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Bringing Computation to the Data: Interoperable Serverless Function Execution for Astrophysical Data Analysis in the SRCNet

  • Parra-Royón, Manuel
  • Garrido, Julián
  • Sánchez-Expósito, S.
  • Mendoza, M. A.
  • Barnsley, Rob
  • Moraghan, Anthony
  • Sánchez, Jesús
  • Darriba, Laura
  • Ruíz-Monje, Carlos
  • Joao, Edgar
  • Moldón, Javier
  • Salgado, Jesús
  • Verdes-Montenegro, Lourdes
Serverless computing is a paradigm in which the underlying infrastructure is fully managed by the provider, enabling applications and services to be executed with elastic resource provisioning and minimal operational overhead. A core model within this paradigm is Function-as-a-Service (FaaS), where lightweight functions are deployed and triggered on demand, scaling seamlessly with workload. FaaS offers flexibility, cost-effectiveness, and fine-grained scalability, qualities particularly relevant for large-scale scientific infrastructures where data volumes are too large to centralise and computation must increasingly occur close to the data. The Square Kilometre Array Observatory (SKAO) exemplifies this challenge. Once operational, it will generate about 700 PB of data products annually, distributed across the SKA Regional Centre Network (SRCNet), a federation of international centres providing storage, computing, and analysis services. In such a context, FaaS offers a natural mechanism to bring computation to the data. We studied the principles of Function-as-a-Service (FaaS) and their application to radio astronomy workflows. To demonstrate feasibility, a Gaussian convolution function was fully implemented and integrated within the SRCNet ecosystem. The methodology included a basic evaluation through performance tests to measure execution latency, comparing traditional computing environments against the FaaS-based approach deployed within the SRCNet distributed data and computing infrastructure. Benchmarks demonstrate that the FaaS framework, supported by an SRCNet IVOA Datalink service, successfully enables transparent function execution at the node closest to the data and user. These results show a measurable reduction in execution latency and inter-node data transfers, significantly enhancing overall computational efficiency within the SRCNet architecture. The SRCNet FaaS model, although currently under development, provides a viable pathway for seamless function deployment across a federated ecosystem. By leveraging SRCNet IVOA Datalink for high-performance, data-proximate computation, the system aligns with the SRCNet vision. However, transitioning to larger-scale workloads will necessitate precise resource sizing and high-availability scaling of the underlying functional nodes. © The Author(s) 2026., Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. MP, SSE, JG, JS, MM, LD, EJ and LVM acknowledge financial support from the grant CEX2021- 001131-S funded by MICIU/AEI/ 10.13039/501100011033 and from the grant TED2021-130231B-I00 funded by MICIU/AEI/ and by the European Union NextGenerationEU/PRTR, acknowledges financial support from the grants PID2021-123930OB-C21 and PID2024- 155817OB-I00 funded by MICIU/AEI/ and by ERDF/EU, and by the grant INFRA24023 (CSIC4SKA) funded by CSIC. Prototype of an SRC (SPSRC) service and support funded by the Ministerio de Cien- cia, Innovación y Universidades (MICIU), by the Junta de Andalucía, by the European Regional Development Fund (ERDF) and by the Eu- ropean Union NextGenerationEU/PRTR. The SPSRC acknowledges financial support from the Agencia Estatal de Investigación (AEI) through the "Center of Excellence Severo Ochoa" award to the Insti- tuto de Astrofísica de Andalucía (IAA-CSIC) (SEV-2017-0709) and from the grant CEX2021–001131-S funded by MICIU/AEI. AM ac- knowledges support from the UK SKA Regional Centre (UKSRC). The UKSRC is a collaboration between the University of Cambridge, University of Edinburgh, Durham University, University of Hertford- shire, University of Manchester, University College London, and the UKRI Science and Technology Facilities Council (STFC) Scientific Computing at RAL. The UKSRC is supported by funding from the UKRI STFC. JG, SSE, and Lourdes Verdes-Montenegro funded this research., With funding from the Spanish government through the "Severo Ochoa Centre of Excelence" accreditation (CEX2021-001131-S), With funding from the Spanish government through the "María de Maeztu Unit of Excellence" accreditation (SEV-2017-0709), Peer reviewed
DIGITAL.CSIC. Repositorio Institucional del CSIC
Publicación Artículo científico (JournalArticle)

Towards a sustainable astronomical data infrastructure: Optimising linking data from the Rucio datalake to the users areas within the SKA Regional Centres Network

  • Parra-Royón, Manuel
  • Garrido, Julián
  • Sánchez-Expósito, S.
  • Darriba-Pol, Laura
  • Sánchez-Castañeda, Jesús
  • Mendoza, M. Ángeles
  • Coles, Jeremy
  • McConkey, Sean
  • Joshi, Rohini
  • Barnsley, Rob
  • Salgado, Jesús
  • Verdes-Montenegro, Lourdes
The distributed architecture of the SKA Regional Centre Network (SRCNet) aims to provide scientific communities worldwide with efficient computational and storage resources to exploit the massive data volumes produced by the SKA Observatory (SKAO). Given the amount of SKAO data, traditional data management paradigms — where data is transferred to computational resources— are no longer feasible. Instead, computational workflows must increasingly be relocated closer to data storage locations, emphasizing efficient data access strategies and avoiding unnecessary duplication or redundancy. In this context, we present PrepareData, a modular and extensible data delivery service developed within SRCNet prototyping activities. Our proposal for this service addresses the critical challenge of redundant data transfers and duplication at both node and user levels by enabling seamless delivery of requested datasets from local Rucio Storage Elements (RSEs) directly into users’ working environments. PrepareData operates as a local service within each SRCNet node and it is integrated into a broader ecosystem of federated services. Specifically, we designed and evaluated two distinct yet complementary implementations to avoid unnecessary data duplication and to enable a dynamic data bridge between the RSEs and the user storage areas, through: (1) a filesystem-based solution leveraging CephFS, which uses shared filesystem mount points and bind mounts to ensure consistent and immediate data availability of the data across computational nodes, and (2) a Kubernetes model using Persistent Volumes and Persistent Volume Claims, dynamically injecting data into a user’s areas. To tackle this work we detail the architectural design and development, the technical implementation, the integration of both solutions with science enabling tools, such as JupyterHub, CARTA or virtually any application, and finally we provide a performance evaluation. This contribution provides a scalable and sustainable blueprint for data delivery in federated scientific infrastructures, supporting the broader goals of green computing and efficient resource utilisation. Copyright: © 2026 Parra-Royón M et al., MP, SSE, JG, JS, MM, LD, EJ and LVM acknowledge financial support from the grant CEX2021–001131-S funded by MICIU/AEI/ 10.13039/501100011033 and from the grant TED2021-130231B-I00 funded by MICIU/AEI/ 10.13039/501100011033 and by the European Union NextGenerationEU/PRTR, acknowledges financial support from the grant PID2021-123930OB-C21, PID2024-155817OB-I00 funded by MICIU/AEI/ 10.13039/501100011033 and by ERDF/EU, and by the grant INFRA24023 (CSIC4SKA) funded by CSIC. MP, JG, SSE, LVM acknowledge support from the European Science Cluster of Astronomy and Particle Physics ESFRI Research Infrastructures project that has received funding from the European Union’s Horizon 2020 research and innovation program under Grant Agreement No. 824064. Prototype of an SRC (SPSRC) service and sup-port funded by the Ministerio de Ciencia, Innovación y Universidades (MICIU), by the Junta de Andalucía, by the European Regional Development Fund (ERDF) and by the European Union NextGenerationEU/PRTR. The SPSRC acknowledges financial support from the Agencia Estatal de Investigación (AEI) through the “Center of Excellence Severo-Ochoa” award to the Instituto de Astrofísica de Andalucía (IAA-CSIC) (SEV-2017-0709) and from the grant CEX2021–001131-S funded by MICIU/AEI/ 10.13039/501100011033., With funding from the Spanish government through the "Severo Ochoa Centre of Excelence" accreditation (CEX2021–001131-S), With funding from the Spanish government through the "Severo Ochoa Centre of Excelence" accreditation (SEV-2017-0709), Peer reviewed