Pasar al contenido principal

IA RESPONSABLE PARA MINERIA DE PROCESOS 2.0

PID2020-112623GB-I00

•
Nombre agencia financiadora Agencia Estatal de Investigación
Acrónimo agencia financiadora AEI
Programa Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Convocatoria Proyectos de I+D+i (Generación de Conocimiento y Retos Investigación)
Año convocatoria 2020
Plan Estatal 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Centro beneficiario UNIVERSIDADE DE SANTIAGO DE COMPOSTELA

Resultados relacionados

Resultados totales (Incluyendo duplicados): 20
Encontrada(s) 1 página(s)

Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Libro o Monografía (Book)

On recursion and its varieties. Formal, Cognitive and Use Aspects of Recursion in Language

  • Sobrino Cerdeiriña, Alejandro
Recursion is presented in this book as a varied and multiform phenomenon, far from the supposed singularity it seemed to have in the field of mathematics or linguistics. Although plural, it also presents similarities, which have to do with a property of the human cognitive system that permeates many of its thoughts and actions. Usually, science tries to find what is general, setting aside what is individual or specific, adapting reality to models. But sometimes it is not possible to find regularities, because, as it is popularly said, reality rebels and makes us notice the opposite: singularity, diversity. Studies on recursion move between these two poles, an idealistic view of what should be and a realistic view of what is found. Both perspectives are analyzed extensively in this text, showing recursion as a species-specific property that is, at the same time, culturally influenced.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

REACH: Researching Efficient Alignment-based Conformance Checking

  • Casas-Ramos, Jacobo
  • Mucientes Molina, Manuel
  • Lama Penín, Manuel
Conformance checking techniques compare how a process is supposed to be executed according to a model with how it is executed in reality according to an event log. Alignment-based approaches are the most successful solutions for conformance checking. Optimal alignments are a way of finding the best match between the real and the modeled behavior and identifying the differences. However, finding these optimal alignments is a challenging task, especially for complex cases where the log and the model have many events and paths. The difficulty lies in the computational complexity required to find these alignments. To address this problem, we propose an efficient algorithm named REACH based on the A* search algorithm. The core components of the proposal are the use of a partial reachability graph for faster execution of process models for alignment computation and a set of optimization techniques for reducing the number of states explored by the A* algorithm. These improve performance by both reducing the required computation time per state and the number of states to process respectively. To evaluate the performance and scalability, we conducted tests using 227 pairs of logs and models, comparing the results obtained with those from 10 state-of-the-art approaches. Results show that REACH outperforms the other proposals in runtimes, and even aligns logs and models that no other algorithm is able to align.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

Enriching interactive explanations with fuzzy temporal constraint networks

  • Canabal Juanatey, Mariña
  • Alonso Moral, José María
  • Catalá Bolós, Alejandro
  • Bugarín-Diz, Alberto
Humans often use expressions with vague terms which play a fundamental role for effective communication. These expressions are successfully modeled with fuzzy technology, but they are not usually integrated yet with Natural Language Processing models and techniques. Large-scale pre-trained language models yield excellent results in many language tasks, but they have some drawbacks such as their lack of transparency and thorough temporal reasoning capabilities. Therefore, the use of such models may provoke inconsistent or incorrect dialogues in the context of conversational agents which were aimed at providing users of intelligent systems with interactive explanations. In this paper, we propose a model for fuzzy temporal reasoning to overcome some inconsistencies detected in pre-trained language models in a specific application domain of a conversational agent carefully designed for providing users with explanations which are endowed with a good balance between naturalness and fidelity. More precisely, starting from a knowledge graph that provides an intuitive representation of the entities and relations in the application domain, we describe how to map the temporal information onto a fuzzy temporal constraint network. This formalism allows to represent imprecise temporal information and provides mechanisms for checking consistency in conversations. In addition, as a proof of concept, we have developed TimeVersa, a conversational agent which integrates the proposed model into an application domain (i.e., a virtual assistant for tourists) that requires handling imprecise temporal constraints. We illustrate in a use case how the agent can identify temporal inconsistencies and answer queries related to temporal information properly. Results after a user study report that users' perception of consistency is significantly higher in a conversation with TimeVersa than in a similar conversation using the well-known GPT-3 Large Language Model, when vague temporal information is involved. The proposed approach is a step forward for developing conversational agents operating in application domains that require temporal reasoning under uncertainty.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Capítulo o parte de un libro (BookPart)

A User Study on the Utility of Context-aware Explanations for Assisting Data Scientists in Error Analysis of Fuzzy Decision Trees

  • Fernández, Guillermo
  • Gámez, Jose A.
  • Puerta, Jose M.
  • Aledo, Juan A.
  • Alonso Moral, José María
  • Bugarín-Diz, Alberto
The current growth of interest in the field of eXplainable Artificial Intelligence has led to the rise of multiple explanation generation techniques to bridge the gap between the algorithmic complexity of the most powerful algorithms and their end users, who are expected to take advantage of them. In addition, explanations can assist with other tasks, such as explaining the algorithm to a model designer, who can use that information to fine-tune the system that implements such an algorithm. This gives the model designer some insight into the inner workings of the system to guide design decisions as well as identify and prevent potential errors, resulting in a myriad of improvements regarding model properties such as accuracy, explainability, trustworthiness, coherence with existing knowledge, etc. In this paper, we introduce a method to enrich local explanations of fuzzy decision tree classifications with context information. The goodness of the proposed method was validated with a user study in which 26 participants had to detect classification errors made by a fuzzy decision tree. Reported results show that enriched explanations allowed the participants to be more accurate in detecting errors in the classification system, albeit taking longer to solve the task with respect to participants who were shown regular local explanations.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Capítulo o parte de un libro (BookPart)

A Causal Model Application to a Cultural Heritage Sentence Analysis

  • Sobrino Cerdeiriña, Alejandro
  • Calderón Cerrato, Beatriz
In this paper we will approach a cultural heritage sentence focusing on its causal content with the aim of providing a causal graph that, once pruned using bayesian techniques, schematically shows in an abbreviated way the essential content of the sentence for a non-specialist or general audience. For that purpose, the paper develops the following story line. We begin by noting the frequent controversies around heritage and its prosecution when discrepancies emerge. Next, we analyze a Spanish legal sentence about cultural heritage focusing on its causal structure and lexicon. In this respect, relevant aspects of causality are discussed both from a logical and a lexical point of view, which makes it possible to extract from the text of the judgment those sentences that are causally most salient. Differences between causality of physical and law facts are also cleared. Finally, a causal graph is depicted from the selected set of causal phrases of the sentence and a Bayesian analysis is applied to separate effective causes from the spurious ones for understanding the judge’s verdict, concluding the usefulness of the causal analysis with the aim to grasp the factual and evidentiary contents of a sentence about heritage.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

A full data augmentation pipeline for small object detection based on generative adversarial networks

  • Bosquet Mera, Brais
  • Cores Costa, Daniel
  • Brea Sánchez, Víctor Manuel
  • Mucientes Molina, Manuel
  • Bimbo, Alberto del
Object detection accuracy on small objects, i.e., objects under 32 32 pixels, lags behind that of large ones. To address this issue, innovative architectures have been designed and new datasets have been released. Still, the number of small objects in many datasets does not suffice for training. The advent of the generative adversarial networks (GANs) opens up a new data augmentation possibility for training architectures without the costly task of annotating huge datasets for small objects. In this paper, we propose a full pipeline for data augmentation for small object detection which combines a GAN-based object generator with techniques of object segmentation, image inpainting, and image blending to achieve high-quality synthetic data. The main component of our pipeline is DS-GAN, a novel GAN-based architecture that generates realistic small objects from larger ones. Experimental results show that our overall data augmentation method improves the performance of state-of-the-art models up to 11.9% AP on UAVDT and by 4.7% AP on iSAID, both for the small objects subset and for a scenario where the number of training instances is limited.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

Real-time siamese multiple object tracker with enhanced proposals

  • Vaquero Otal, Lorenzo
  • Brea Sánchez, Víctor Manuel
  • Mucientes Molina, Manuel
Maintaining the identity of multiple objects in real-time video is a challenging task, as it is not always feasible to run a detector on every frame. Thus, motion estimation systems are often employed, which either do not scale well with the number of targets or produce features with limited semantic information. To solve the aforementioned problems and allow the tracking of dozens of arbitrary objects in real-time, we propose SiamMOTION. SiamMOTION includes a novel proposal engine that produces quality features through an attention mechanism and a region-of-interest extractor fed by an inertia module and powered by a feature pyramid network. Finally, the extracted tensors enter a comparison head that efficiently matches pairs of exemplars and search areas, generating quality predictions via a pairwise depthwise region proposal network and a multi-object penalization module. SiamMOTION has been validated on five public benchmarks, achieving leading performance against current state-of-the-art trackers. Code available at: https://www.github.com/lorenzovaquero/SiamMOTION
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

On the role of explanations in machine learning prediction of nonalcoholic steatohepatitis

  • Babakov, Nikolay
  • Rezgova, Elena
  • Reiter, Ehud
  • Bugarín-Diz, Alberto
Nonalcoholic steatohepatitis (NASH) is a common disease that ultimately can lead to the development of end-stage liver disease, cirrhosis, or hepatocellular carcinoma. An early prediction of NASH provides an opportunity to make an appropriate strategy for prevention, early diagnosis, and treatment. The most accurate approach for NASH diagnostics is a liver biopsy, which can lead to various complications for the patient. Many papers have studied non-invasive machine learning (ML)-driven approaches to early non-invasive NASH prediction; however, to the best of our knowledge, none of the works considered the problem of explainability of the trained ML models to the medical experts. In this work, we address this issue. We use the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) nonalcoholic fatty liver disease adult database to train different ML models and propose the technique to explain their predictions. We compare the explanations obtained from a transparent model (Decision Tree) and a non-transparent model (Random Forest). Furthermore, we analyze the quality of explanation prediction by objective means and with a user study involving 11 medical practitioners. Our findings show that there is no significant difference in the perception of explanation obtained from transparent and non-transparent models, and that the explanation of the models’ predictions slightly increases their usability and trustworthiness for real practitioners, enhancing their practical adoption in clinical settings.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

Enhancing few-shot object detection through pseudo-label mining

  • García Fernández, Pablo
  • Cores Costa, Daniel
  • Mucientes Molina, Manuel
Few-shot object detection involves adapting an existing detector to a set of unseen categories with few annotated examples. This data limitation makes these methods to underperform those trained on large labeled datasets. In many scenarios, there is a high amount of unlabeled data that is never exploited. Thus, we propose to exPAND the initial novel set by mining pseudo-labels. From a raw set of detections, xPAND obtains reliable pseudo-labels suitable for training any detector. To this end, we propose two new modules: Class and Box confirmation. Class Confirmation aims to remove misclassified pseudo-labels by comparing candidates with expected class prototypes. Box Confirmation estimates IoU to discard inadequately framed objects. Experimental results demonstrate that xPAND enhances the performance of multiple detectors up to +5.9 nAP and +16.4 nAP50 points for MS-COCO and PASCAL VOC, respectively, establishing a new state of the art. Code: https://github.com/PAGF188/xPAND
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Capítulo o parte de un libro (BookPart)

Few-Shot Image Classification for Automatic COVID-19 Diagnosis

  • Cores Costa, Daniel
  • Vila Blanco, Nicolás
  • Mucientes Molina, Manuel
  • Carreira Nouche, María José
Developing robust and performant methods for diagnosing COVID-19, particularly for triaging processes, is crucial. This study introduces a completely automated system to detect COVID-19 by means of the analysis of Chest X-Ray scans (CXR). The proposed methodology is based on few-shot techniques, enabling to work on small image datasets. Moreover, a set of additions have been done to enhance the diagnostic capabilities. First, a network to extract the lung region to rely only on the most relevant image area. Second, a new cost function to penalize each misclassification according to the clinical consequences. Third, a system to combine different predictions from the same image to increase the robustness of the diagnoses. The proposed approach was validated on the public dataset COVIDGR-1.0, yielding a classification accuracy of 79.10% ± 3.41% and, thus, outperforming other state-of-the-art methods. In conclusion, the proposed methodology has proven to be suitable for the diagnosis of COVID-19.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

An Empirical Study on the Number of Items in Human Evaluation of Automatically Generated Texts

  • González-Corbelle, Javier
  • Alonso Moral, José María
  • Crujeiras Casais, Rosa María
  • Bugarín-Diz, Alberto
Human evaluation of neural models in the research field of Natural Language Generation (NLG) requires a careful experimental design (e.g., number of evaluators, number of items to assess, number of quality criteria, etc.) for the sake of reproducibility as well as for ensuring that it is possible to draw significant conclusions. Although there are some generic recommendations on how to proceed, there is not an evaluation protocol admitted worldwide. In this paper, we address empirically the impact of the number of items to assess in the context of the human evaluation of NLG systems. We apply different resampling methods to simulate the evaluation of different sets of items by each evaluator. Then, we compare the results obtained by evaluating only a limited set of items with those obtained by evaluating all outputs of the system for a given test set. Empirical findings validate the initial research hypothesis: well-known resampling statistical methods can contribute to getting significant results even with a small number of items to be evaluated by each evaluator.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

An empirically supported approach to the treatment of imprecision in vague reasoning

  • Velasco Benito, Gael
  • Sobrino Cerdeiriña, Alejandro
  • Bugarín-Diz, Alberto
The aim of this paper is to propose a new approach for the automatic treatment of linguistic vagueness. Our motivation is the feeling that most existing approaches dealing with linguistic information are based on converting vague meaning into crisp meaning using some conversions to precise measurements. As a result, existing approaches are adequate and easy to implement, but do not closely model the human thought process. To help alleviate this deficiency, we propose the use of linguistic relations to provide a natural language interface to an end user. We show a possible linguistic Prolog model based on an extension of the syntactic unification algorithm using synonymy and antonymy, as well as the extension of the resolution principle. Our approach does not aim to provide a well-founded formal semantics for such a linguistic Prolog, but a simple model supported by two experiments focused on the use of vague language, both of them executed in Spanish (an analysis of the data of the first experiment it is also available in that language at [1]). Thus, the purpose of this paper is to contribute to the mechanization of approximate reasoning by being respectful of the semantics of the vague terms involved in it; i.e., by paying attention to how they are evaluated by linguistic users under experimentation
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Capítulo o parte de un libro (BookPart)

MixUDA: From Synthetic to Real Object Detection

  • Gil Pérez, Pablo
  • Cores Costa, Daniel
  • Mucientes Molina, Manuel
Object detection has made remarkable progress in recent years, driven by advancements in deep learning and the availability of large-scale annotated datasets. However, these methods often require extensive labeled data, which may not be accessible for specific or emerging applications. This limitation has generated interest in Unsupervised Domain Adaptation (UDA), which facilitates knowledge transfer from a labeled source domain to an unlabeled and differently distributed target domain. This study addresses the challenge of UDA between synthetic and realworld data. A methodology for generating synthetic datasets is proposed using AirSim and Unreal Engine, enabling the creation of highly customizable and diverse datasets. We also propose a Domain Adaptation technique, MixUDA, that maximizes the utility of the synthetic dataset to improve the performance of a model in a real domain. MixUDA is a UDA approach which uses a Mean Teacher architecture and employs pseudo-labels combined with two different image-mixing operations to achieve a smooth and progressive transition from the synthetic to the real domain: pseudo-mosaic and pseudo-mixup. The obtained results demonstrate encouraging progress, as MixUDA surpasses state-of-the-art models D3T and MixPL by 1.18 and 4 AP points respectively, approaching performance of oracle models trained directly on the target domain. These findings suggest that synthetic datasets have significant potential in addressing data scarcity and improving model generalization, while also pointing to promising directions for further exploration in this area.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

Trustworthy Artificial Intelligence in Alzheimer’s Disease: State of the Art, Opportunities, and Challenges

  • El-Sappagh, Shaker
  • Alonso Moral, José María
  • Abuhmed, Tamer
  • Ali, Farman
  • Bugarín-Diz, Alberto
Medical applications of Artificial Intelligence (AI) have consistently shown remarkable performance in providing medical professionals and patients with support for complex tasks. Nevertheless, the use of these applications in sensitive clinical domains where high-stakes decisions are involved could be much more extensive if patients, medical professionals, and regulators were provided with mechanisms for trusting the results provided by AI systems. A key issue for achieving this is endowing AI systems with key dimensions of Trustworthy AI (TAI), such as fairness, transparency, robustness, or accountability, which are not usually considered within this context in a generalized and systematic manner. This paper reviews the recent advances in the TAI domain, including TAI standards and guidelines. We propose several requirements to be addressed in the design, development, and deployment of TAI systems and present a novel machine learning pipeline that contains TAI requirements as embedded components. Moreover, as an example of how current AI systems in medicine consider the TAI perspective, the study extensively reviews the recent literature (2017-2021) on AI systems in a prevalent and high social-impact disease: diagnosis and progression detection of Alzheimer’s Disease (AD). The most relevant AI systems in the AD domain are compared and discussed (such as machine learning, deep learning, ensembles, time series, and multimodal multitask) from the perspective of how they address TAI in their design. Several open challenges are highlighted, which could be claimed as one of the main reasons to justify the rare application of AI systems in real clinical environments. The study provides a roadmap to measure the TAI status of an AI systems and highlights its limitations. In addition, it provides the main guidelines to overcome these limitations and build medically trusted AI-based applications in the medical domain.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

Automatic generation of textual descriptions in data-to-text systems using a fuzzy temporal ontology: Application in air quality index data series

  • Cascallar Fuentes, Andrea
  • Gallego Fernández, Javier
  • Ramos Soto, Alejandro
  • Saunders, Anthony
  • Bugarín-Diz, Alberto
In this paper we present a model based on computational intelligence and natural language generation for the automatic generation of textual summaries from numerical data series, aiming to provide insights which help users to understand the relevant information hidden in the data. Our model includes a fuzzy temporal ontology with temporal references which addresses the problem of managing imprecise temporal knowledge, which is relevant in data series. We fully describe a real use case of application in the environmental information systems field, providing linguistic descriptions about the air quality index (AQI), which is a very well-known indicator provided by all meteorological agencies worldwide. We consider two different data sources of real AQI data provided by the official Galician (NW Spain) Meteorology Agency: (i) AQI distribution in the stations of the meteorological observation network and (ii) time series which describe the state and evolution of the AQI in each meteorological station. Both application models were evaluated following the current standards and good practices of manual human expert evaluation of the Natural Language Generation field. Assessment results by two experts meteorologists were very satisfactory, which empirically confirm that the proposed textual descriptions fit this type of data and service both in content and layout
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

VERONA: A python library for benchmarking deep learning in business process monitoring

  • Gamallo Fernández, Pedro
  • Rama Maneiro, Efrén
  • Vidal Aguiar, Juan Carlos
  • Lama Penín, Manuel
Predictive process monitoring is a subfield of process mining that focuses on predicting the future behavior of real-world processes, anticipating constraint violations and bottlenecks, and enabling real-time decision making. Among other machine learning approaches, Deep Learning-based architectures have achieved high levels of prediction accuracy, becoming an increasingly prolific area of research in recent years. However, the variety of datasets, learning techniques, and metrics used makes the comparison of proposals complicated and biased. To address this problem this paper presents VERONA, a Python library designed for the development of the deep learning predictive process monitoring pipeline. Additionally, this library provides a framework for replicating the experimental setup of the state-of-the-art benchmark in the field, enabling streamlined comparison of new approaches and improving the reproducibility of experiments.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

Meta-heuristics for generation of linguistic descriptions of weather data: experimental comparison of two approaches

  • Cascallar Fuentes, Andrea
  • Ramos Soto, Alejandro
  • Bugarín-Diz, Alberto
In this paper we experimentally assess, from both algorithmic and pragmatic perspectives, the adequacy of linguistic descriptions of real data generated by two metaheuristics: simulated annealing and genetic algorithm meta-heuristics. The type of descriptions we consider are fuzzy quantified statements (both Zadeh's type-1 and type-2) involving three well-known quantification models (Zadeh's scalar and fuzzy and Delgado's GD). We conducted an empirical validation using real observation and prediction meteorological data, where both automatic (metrics-based) and manual (human experts-based) assessment on the adequacy of the generated descriptions was assessed. Results indicate that, overall, the genetic approach performs better than simulated annealing in terms of quality of the obtained descriptions and time execution. Significance of this outperforming depends on the type of meteorological data and the quantification model selected. Tests of statistical significance point out that for type-1 descriptions no significant differences exist between the two meta-heuristics in the prediction case. For type-2 descriptions, significant differences exist for Delgado's GD model for both types of data. For Zadeh's scalar and fuzzy quantification significance depends on the type of data (observation or prediction). Globally, outperforming of the genetic approach over simulated annealing i) is significant in 4 out of 12 scenarios considered (all of them type-2), and ii) is not significant in the other 8 out 12 scenarios (all type-1 and two type-2). Also human expert assessment on the adequacy of the descriptions was conducted, showing that both meta-heuristics behave similarly for type-1 descriptions, while genetic algorithms produce more suitable type-2 linguistic descriptions
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Capítulo o parte de un libro (BookPart)

Virtual Assistant supported by Fusion of Fuzzy Rule-based Explainers

  • Pérez-Ferreiro, Pablo Miguel
  • Catalá Bolós, Alejandro
  • Bugarín-Diz, Alberto
  • Alonso Moral, José María
In this work, we introduce a novel methodology for combining different explanatory inputs into a singular coherent narrative in the medical domain. As proof of concept, we have implemented a prototype ready to diagnose diabetes and heart disease. The proposal is validated with a user study in which we collected 48 valid responses to a given questionnaire. We compared four different strategies for explanation fusion, considering explanations coming from two different fuzzy rule bases. The winner strategy (i.e., verbalizing common antecedents as the main factors of the diagnosis and adding any non-common antecedents as supporting evidence) was perceived as trustworthy, useful, and natural by most participants in the study.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Artículo científico (JournalArticle)

A fine-tuning approach based on spatio-temporal features for few-shot video object detection

  • Cores Costa, Daniel
  • Seidenari, Lorenzo
  • Bimbo, Alberto del
  • Brea Sánchez, Víctor Manuel
  • Mucientes Molina, Manuel
This paper describes a new Fine-Tuning approach for Few-Shot object detection in Videos that exploits spatio-temporal information to boost detection precision. Despite the progress made in the single image domain in recent years, the few-shot video object detection problem remains almost unexplored. A few-shot detector must quickly adapt to a new domain with a limited number of annotations per category. Therefore, it is not possible to include videos in the training set, hindering the spatio-temporal learning process. We propose augmenting each training image with synthetic frames to train the spatio-temporal module of our method. This module employs attention mechanisms to mine relationships between proposals across frames, effectively leveraging spatio-temporal information. A spatio-temporal double head then localizes objects in the current frame while classifying them using both context from nearby frames and information from the current frame. Finally, the predicted scores are fed into a long-term object-linking method that generates object tubes across the video. By optimizing the classification score based on these tubes, our approach ensures spatio-temporal consistency. Classification is the primary challenge in few-shot object detection. Our results show that spatio-temporal information helps to mitigate this issue, paving the way for future research in this direction. FTFSVid achieves 41.9 AP50 on the Few-Shot Video Object Detection (FSVOD-500) and 42.9 AP50 on the Few-Shot YouTube Video (FSYTV-40) dataset, surpassing our spatial baseline by 4.3 and 2.5 points. Additionally, FTFSVid outperforms previous few-shot video object detectors by 3.2 points on FSVOD-500 and 14.5 points on FSYTV-40, setting a new state-of-the-art.
Minerva. Repositorio Institucional de la Universidad de Santiago de Compostela
Publicación Capítulo o parte de un libro (BookPart)

Exploring Open-Vocabulary Models for Category-Free Detection

  • García Fernández, Pablo
  • Mucientes Molina, Manuel
  • Cores Costa, Daniel
Object detection models typically rely on a predefined setof categories, limiting their applicability in real-world scenarios whereobject classes may be unknown. In this paper, we propose a novel,training-free framework that enables off-the-shelf open-vocabulary ob-ject detectors (OvOD) to perform category-free detection —localizingand classifying objects without any prior category knowledge. Our ap-proach leverages image captioning to dynamically generate descriptiveterms directly from the image content, followed by a WordNet-based fil-tering process to extract semantically meaningful category names. Thesediscovered categories are then embedded and matched with visual regionfeatures using a frozen OvOD model to perform detection. We evaluateour method on the COCO dataset in a fully zero-shot setting and demon-strate that it significantly outperforms strong multimodal large languagemodel baselines, achieving an improvement of over 30 AP points. Thishighlights our method as a promising direction for more adaptive solu-tions to real-world detection challenges.