Skip to main content

DESARROLLO DE AGENTES DE NAVEGACION AUTONOMOS

TIN2017-88709-R

•
Funding agency name Agencia Estatal de Investigación
Funding agency acronym AEI
Program Programa Estatal de I+D+i Orientado a los Retos de la Sociedad
Call Proyectos I+D+i Retos de Investigación
Call year 2017
State Plan 2017-2020 Plan Estatal de Investigación Científica y Técnica y de Innovación
Beneficiary centre CENTRO DE VISION POR COMPUTADOR

Resultados relacionados

Resultados totales (Incluyendo duplicados): 22
Encontrada(s) 1 página(s)

RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Variable Rate Deep Image Compression With Modulated Autoencoder

  • Yang, Fei
  • Herranz, Luis
  • Van de Weijer, Joost
  • Iglesias-Guitian, Jose A.
  • López, Antonio M.
  • Mozerov, Mikhail G.
This version of the paper has been accepted for publication. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., [Abstract]: Variable rate is a requirement for flexible and adaptable image and video compression. However, deep image compression methods (DIC) are optimized for a single fixed rate-distortion (R-D) tradeoff. While this can be addressed by training multiple models for different tradeoffs, the memory requirements increase proportionally to the number of models. Scaling the bottleneck representation of a shared autoencoder can provide variable rate compression with a single shared autoencoder. However, the R-D performance using this simple mechanism degrades in low bitrates, and also shrinks the effective range of bitrates. To address these limitations, we formulate the problem of variable R-D optimization for DIC, and propose modulated autoencoders (MAEs), where the representations of a shared autoencoder are adapted to the specific R-D tradeoff via a modulation network. Jointly training this modulated autoencoder and the modulation network provides an effective way to navigate the R-D operational curve. Our experiments show that the proposed method can achieve almost the same R-D performance of independent models with significantly fewer parameters, This work was supported in part by Audi Electronics Venture GmbH. The Computer Vision Center (CVC) was supported by the Generalitat de Catalunya CERCA Program and its ACCIO agency. The work of F. Yang was supported by the Chinese Scholarship Council under Grant 201706290127. The work of L. Herranz was supported by the Spanish project RTI2018-102285-A-I00. The work of J. van de Weijer was supported by the Spanish project TIN2016-79717-R. The work of A. M. López and J. A. I. Guitián was supported by the project TIN2017-88709-R (MINECO/AEI/FEDER, UE). The work of A. M. López was also supported by the ICREA Academia programme. The work of J. A. I. Guitián and L. Herranz was supported by the EU’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie Grant 665919
RUC. Repositorio da Universidade da Coruña
Publication Artículo científico (JournalArticle)

Real-Time Denoising of Volumetric Path Tracing for Direct Volume Rendering

  • Iglesias-Guitian, Jose A.
  • Mane, Prajita
  • Moon, Bochang
© 2020 IEEE. This version of the article has been accepted for publication, after peer review. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The Version of Record is available online at: https://doi.org/10.1109/TVCG.2020.3037680, [Abstract]: Direct volume rendering (DVR) using volumetric path tracing (VPT) is a scientific visualization technique that simulates light transport with objects’ matter using physically-based lighting models. Monte Carlo (MC) path tracing is often used with surface models, yet its application for volumetric models is difficult due to the complexity of integrating MC light-paths in volumetric media with none or smooth material boundaries. Moreover, auxiliary geometry-buffers (G-buffers) produced for volumes are typically very noisy, failing to guide image denoisers relying on that information to preserve image details. This makes existing real-time denoisers, which take noise-free G-buffers as their input, less effective when denoising VPT images. We propose the necessary modifications to an image-based denoiser previously used when rendering surface models, and demonstrate effective denoising of VPT images. In particular, our denoising exploits temporal coherence between frames, without relying on noise-free G-buffers, which has been a common assumption of existing denoisers for surface-models. Our technique preserves high-frequency details through a weighted recursive least squares that handles heterogeneous noise for volumetric models. We show for various real data sets that our method improves the visual fidelity and temporal stability of VPT during classic DVR operations such as camera movements, modifications of the light sources, and editions to the volume transfer function, The authors would like to thank reviewers for their insightful feedback. We also thank S. Oh, F. Lumbreras and J. Serrat for their help with Tensorflow, and T. Kroes for Exposure Render. The datasets are courtesy of the OsiriX Foundation (MANIX), University of Texas (CHAMELEON), University of Arizona (HELODERMA) and XYZ RGB Inc. (DRAGON). This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie g.a. No 665919. J.A. Iglesias-Guitian acknowledges the UDC-Inditex InTalent programme, the Spanish Ministry project TIN2017-88709-R, RYC2018-025385-I (MCIU/AEI/FEDER, EU), FEDER Galicia ED431G 2019/01 and the Nvidia GPU Grant Program. B. Moon was supported by The Cross-Ministry Giga KOREA Project grant funded by the Korea government (MSIT) (No. GK20P0300), Xunta de Galicia; ED431G 2019/01, Korea. Korea government (MSIT); GK20P0300
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Slanted Stixels, A Way to Represent Steep Streets

  • Hernández Juárez, Daniel|||0000-0001-5878-1549
  • Schneider, Lukas
  • Cebrian, Pau|||0000-0002-0650-4117
  • Espinosa, Antonio|||0000-0002-6460-3789
  • Vázquez Bermúdez, David|||0000-0002-2845-8158
  • López, Antonio
  • Franke, Uwe
  • Pollefeys, Marc
  • Moure, Juan C.|||0000-0001-6697-0331
This work presents and evaluates a novel compact scene representation based on Stixels that infers geometric and semantic information. Our approach overcomes the previous rather restrictive geometric assumptions for Stixels by introducing a novel depth model to account for non-flat roads and slanted objects. Both semantic and depth cues are used jointly to infer the scene representation in a sound global energy minimization formulation. Furthermore, a novel approximation scheme is introduced in order to significantly reduce the computational complexity of the Stixel algorithm, and then achieve real-time computation capabilities. The idea is to first perform an over-segmentation of the image, discarding the unlikely Stixel cuts, and apply the algorithm only on the remaining Stixel cuts. This work presents a novel over-segmentation strategy based on a fully convolutional network, which outperforms an approach based on using local extrema of the disparity map. We evaluate the proposed methods in terms of semantic and geometric accuracy as well as run-time on four publicly available benchmark datasets. Our approach maintains accuracy on flat road scene datasets while improving substantially on a novel non-flat road dataset.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Recognizing New Classes with Synthetic Data in the Loop, Application to Traffic Sign Recognition

  • Villalonga, Gabriel|||0000-0002-1155-9374
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
On-board vision systems may need to increase the number of classes that can be recognized in a relatively short period. For instance, a traffic sign recognition system may suddenly be required to recognize new signs. Since collecting and annotating samples of such new classes may need more time than we wish, especially for uncommon signs, we propose a method to generate these samples by combining synthetic images and Generative Adversarial Network (GAN) technology. In particular, the GAN is trained on synthetic and real-world samples from known classes to perform synthetic-to-real domain adaptation, but applied to synthetic samples of the new classes. Using the Tsinghua dataset with a synthetic counterpart, SYNTHIA-TS, we have run an extensive set of experiments. The results show that the proposed method is indeed effective, provided that we use a proper Convolutional Neural Network (CNN) to perform the traffic sign recognition (classification) task as well as a proper GAN to transform the synthetic images. Here, a ResNet101-based classifier and domain adaptation based on CycleGAN performed extremely well for a ratio for new/known classes; even for more challenging ratios such as , the results are also very positive.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Co-Training for Deep Object Detection, Comparing Single-Modal and Multi-Modal Approaches

  • Gomez Zurita, Jose Luis|||0000-0001-9511-1915
  • Villalonga, Gabriel|||0000-0002-1155-9374
  • López Peña, Antonio M.|||0000-0002-6979-5783
Top-performing computer vision models are powered by convolutional neural networks (CNNs). Training an accurate CNN highly depends on both the raw sensor data and their associated ground truth (GT). Collecting such GT is usually done through human labeling, which is time-consuming and does not scale as we wish. This data-labeling bottleneck may be intensified due to domain shifts among image sensors, which could force per-sensor data labeling. In this paper, we focus on the use of co-training, a semi-supervised learning (SSL) method, for obtaining self-labeled object bounding boxes (BBs), i.e., the GT to train deep object detectors. In particular, we assess the goodness of multi-modal co-training by relying on two different views of an image, namely, appearance (RGB) and estimated depth (D). Moreover, we compare appearance-based single-modal co-training with multi-modal. Our results suggest that in a standard SSL setting (no domain shift, a few human-labeled data) and under virtual-to-real domain shift (many virtual-world labeled data, no human-labeled data) multi-modal co-training outperforms single-modal. In the latter case, by performing GAN-based domain translation both co-training modalities are on par, at least when using an off-the-shelf depth estimation model not specifically trained on the translated images.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

End-to-end driving via conditional imitation learning

  • Codevilla Moraes, Felipe|||0000-0002-9815-6841
  • Miiller, Matthias
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Koltun, Vladlen
  • Dosovitskiy, Alexey
Deep networks trained on demonstrations of human driving have learned to follow roads and avoid obstacles. However, driving policies trained via imitation learning cannot be controlled at test time. A vehicle trained end-to-end to imitate an expert cannot be guided to take a specific turn at an upcoming intersection. This limits the utility of such systems. We propose to condition imitation learning on high-level command input. At test time, the learned driving policy functions as a chauffeur that handles sensorimotor coordination but continues to respond to navigational commands. We evaluate different architectures for conditional imitation learning in vision-based driving. We conduct experiments in realistic three-dimensional simulations of urban driving and on a 1/5 scale robotic truck that is trained to drive in a residential area. Both systems drive based on visual input yet remain responsive to high-level navigational commands.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Exploring the limitations of behavior cloning for autonomous driving

  • Codevilla Moraes, Felipe|||0000-0002-9815-6841
  • Santana, Eder
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Gaidon, Adrien
Driving requires reacting to a wide variety of complex environment conditions and agent behaviors. Explicitly modeling each possible scenario is unrealistic. In contrast, imitation learning can, in theory, leverage data from large fleets of human-driven cars. Behavior cloning in particular has been successfully used to learn simple visuomotor policies end-to-end, but scaling to the full spectrum of driving behaviors remains an unsolved problem. In this paper, we propose a new benchmark to experimentally investigate the scalability and limitations of behavior cloning. We show that behavior cloning leads to state-of-the-art results, executing complex lateral and longitudinal maneuvers, even in unseen environments, without being explicitly programmed to do so. However, we confirm some limitations of the behavior cloning approach: Some well-known limitations (e.g., dataset bias and overfitting), new generalization issues (e.g., dynamic objects and the lack of a causal modeling), and training instabilities, all requiring further research before behavior cloning can graduate to real-world driving. The code, dataset, benchmark, and agent studied in this paper can be found at https://github.com/felipecode/coiltraine.
Dipòsit Digital de Documents de la UAB
Publication

Metric learning for novelty and anomaly detection

  • Masana, Marc|||0000-0003-3254-3096
  • Ruiz López, Idoia|||0000-0003-0529-0871
  • Serrat, Joan|||0000-0002-4554-199X
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
When neural networks process images which do not resemble the distribution seen during training, so called out-of-distribution images, they often make wrong predictions, and do so too confidently. The capability to detect out-of-distribution images is therefore crucial for many real-world applications. We divide out-of-distribution detection between novelty detection -images of classes which are not in the training set but are related to those-, and anomaly detection -images with classes which are unrelated to the training set. By related we mean they contain the same type of objects, like digits in MNIST and SVHN. Most existing work has focused on anomaly detection, and has addressed this problem considering networks trained with the cross-entropy loss. Differently from them, we propose to use metric learning which does not have the drawback of the softmax layer (inherent to cross-entropy methods), which forces the network to divide its prediction power over the learned classes. We perform extensive experiments and evaluate both novelty and anomaly detection, even in a relevant application such as traffic sign recognition, obtaining comparable or better results than previous works.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Is the pedestrian going to cross?, answering by 2D pose estimation

  • Fang, Zhijie|||0000-0001-6184-8548
  • López Peña, Antonio M.|||0000-0002-6979-5783
Our recent work suggests that, thanks to nowadays powerful CNNs, image-based 2D pose estimation is a promising cue for determining pedestrian intentions such as crossing the road in the path of the ego-vehicle, stopping before entering the road, and starting to walk or bending towards the road. This statement is based on the results obtained on non-naturalistic sequences (Daimler dataset), i.e. in sequences choreographed specifically for performing the study. Fortunately, a new publicly available dataset (JAAD) has appeared recently to allow developing methods for detecting pedestrian intentions in naturalistic driving conditions; more specifically, for addressing the relevant question is the pedestrian going to cross? Accordingly, in this paper we use JAAD to assess the usefulness of 2D pose estimation for answering such a question. We combine CNN-based pedestrian detection, tracking and pose estimation to predict the crossing action from monocular images. Overall, the proposed pipeline provides new state-of-the-art results.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Training a binary weight object detector by knowledge transfer for autonomous driving

  • Xu, Jiaolong
  • Nie, Yiming
  • Wang, Peng
  • López Peña, Antonio M.|||0000-0002-6979-5783
Autonomous driving has harsh requirements of small model size and energy efficiency, in order to enable the embedded system to achieve real-time on-board object detection. Recent deep convolutional neural network based object detectors have achieved state-of-the-art accuracy. However, such models are trained with numerous parameters and their high computational costs and large storage prohibit the deployment to memory and computation resource limited systems. Low-precision neural networks are popular techniques for reducing the computation requirements and memory footprint. Among them, binary weight neural networks (BWNs) are the extreme case which quantizes the float-point into just 1 bit. BWNs are difficult to train and suffer from accuracy deprecation due to the extreme low-bit representation. To address this problem, we propose a knowledge transfer (KT) method to aid the training of BWN using a full-precision teacher network. We built DarkNet- and MobileNet-based binary weight YOLO-v2 detectors and conduct experiments on KITTI benchmark for car, pedestrian and cyclist detection. The experimental results show that the proposed method maintains high detection accuracy while reducing the model size of DarkNet-YOLO from 257 MB to 8.8 MB and MobileNet-YOLO from 193 MB to 7.9 MB.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Active learning for deep detection neural networks

  • Aghdam, Hamed H.
  • Gonzalez Garcia, Abel
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
The cost of drawing object bounding boxes (i.e. labeling) for millions of images is prohibitively high. For instance, labeling pedestrians in a regular urban image could take 35 seconds on average. Active learning aims to reduce the cost of labeling by selecting only those images that are informative to improve the detection network accuracy. In this paper, we propose a method to perform active learning of object detectors based on convolutional neural networks. We propose a new image-level scoring process to rank unlabeled images for their automatic selection, which clearly outperforms classical scores. The proposed method can be applied to videos and sets of still images. In the former case, temporal selection rules can complement our scoring process. As a relevant use case, we extensively study the performance of our method on the task of pedestrian detection. Overall, the experiments show that the proposed method performs better than random selection.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Co-Training for On-Board Deep Object Detection

  • Villalonga, Gabriel|||0000-0002-1155-9374
  • López Peña, Antonio M.|||0000-0002-6979-5783
Providing ground truth supervision to train visual models has been a bottleneck over the years, exacerbated by domain shifts which degenerate the performance of such models. This was the case when visual tasks relied on handcrafted features and shallow machine learning and, despite its unprecedented performance gains, the problem remains open within the deep learning paradigm due to its data-hungry nature. Best performing deep vision-based object detectors are trained in a supervised manner by relying on human-labeled bounding boxes which localize class instances (i.e. objects) within the training images. Thus, object detection is one of such tasks for which human labeling is a major bottleneck. In this article, we assess co-training as a semi-supervised learning method for self-labeling objects in unlabeled images, so reducing the human-labeling effort for developing deep object detectors. Our study pays special attention to a scenario involving domain shift; in particular, when we have automatically generated virtual-world images with object bounding boxes and we have real-world images which are unlabeled. Moreover, we are particularly interested in using co-training for deep object detection in the context of driver assistance systems and/or self-driving vehicles. Thus, using well-established datasets and protocols for object detection in these application contexts, we will show how co-training is a paradigm worth to pursue for alleviating object labeling, working both alone and together with task-agnostic domain adaptation.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

On offline evaluation of vision-based driving models

  • Codevilla Moraes, Felipe|||0000-0002-9815-6841
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Koltun, Vladlen
  • Dosovitskiy, Alexey
Autonomous driving models should ideally be evaluated by deploying them on a fleet of physical vehicles in the real world. Unfortunately, this approach is not practical for the vast majority of researchers. An attractive alternative is to evaluate models offline, on a pre-collected validation dataset with ground truth annotation. In this paper, we investigate the relation between various online and offline metrics for evaluation of autonomous driving models. We find that offline prediction error is not necessarily correlated with driving quality, and two models with identical prediction error can differ dramatically in their driving performance. We show that the correlation of offline evaluation with driving quality can be significantly improved by selecting an appropriate validation dataset and suitable offline metrics.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Rotate your networks, better weight consolidation and less catastrophic forgetting

  • Liu, Xialei
  • Masana, Marc|||0000-0003-3254-3096
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Bagdanov, Andrew|||0000-0001-6408-7043
In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to the state-of-the-art in lifelong learning without forgetting.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Intention recognition of pedestrians and cyclists by 2D pose estimation

  • Fang, Zhijie|||0000-0001-6184-8548
  • López Peña, Antonio M.|||0000-0002-6979-5783
Anticipating the intentions of vulnerable road users (VRUs) such as pedestrians and cyclists is critical for performing safe and comfortable driving maneuvers. This is the case for human driving and, thus, should be taken into account by systems providing any level of driving assistance, from advanced driver assistant systems (ADAS) to fully autonomous vehicles (AVs). In this paper, we show how the latest advances on monocular vision-based human pose estimation, i.e. those relying on deep Convolutional Neural Networks (CNNs), enable to recognize the intentions of such VRUs. In the case of cyclists, we assume that they follow traffic rules to indicate future maneuvers with arm signals. In the case of pedestrians, no indications can be assumed. Instead, we hypothesize that the walking pattern of a pedestrian allows to determine if he/she has the intention of crossing the road in the path of the ego-vehicle, so that the ego-vehicle must maneuver accordingly (e.g. slowing down or stopping). In this paper, we show how the same methodology can be used for recognizing pedestrians and cyclists' intentions. For pedestrians, we perform experiments on the JAAD dataset. For cyclists, we did not found an analogous dataset, thus, we created our own one by acquiring and annotating videos which we share with the research community. Overall, the proposed pipeline provides new state-of-the-art results on the intention recognition of VRUs.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Distributed learning and inference with compressed images

  • Katakol, Sudeep|||0000-0002-4421-9819
  • Elbarashy, Basem
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
Modern computer vision requires processing large amounts of data, both while training the model and/or during inference, once the model is deployed. Scenarios where images are captured and processed in physically separated locations are increasingly common (e.g. autonomous vehicles, cloud computing, smartphones). In addition, many devices suffer from limited resources to store or transmit data (e.g. storage space, channel capacity). In these scenarios, lossy image compression plays a crucial role to effectively increase the number of images collected under such constraints. However, lossy compression entails some undesired degradation of the data that may harm the performance of the downstream analysis task at hand, since important semantic information may be lost in the process. Moreover, we may only have compressed images at training time but are able to use original images at inference time (i.e. test), or vice versa, and in such a case, the downstream model suffers from covariate shift. In this paper, we analyze this phenomenon, with a special focus on vision-based perception for autonomous driving as a paradigmatic scenario. We see that loss of semantic information and covariate shift do indeed exist, resulting in a drop in performance that depends on the compression rate. In order to address the problem, we propose dataset restoration, based on image restoration with generative adversarial networks (GANs). Our method is agnostic to both the particular image compression method and the downstream task; and has the advantage of not adding additional cost to the deployed models, which is particularly important in resource-limited devices. The presented experiments focus on semantic segmentation as a challenging use case, cover a broad range of compression rates and diverse datasets, and show how our method is able to significantly alleviate the negative effects of compression on the downstream visual task.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Generating human action videos by coupling 3D game engines and probabilistic graphical models

  • de Souza, César Roberto
  • Gaidon, Adrien
  • Cabon, Yohann
  • Murray, Naila
  • López Peña, Antonio M.|||0000-0002-6979-5783
Deep video action recognition models have been highly successful in recent years but require large quantities of manually-annotated data, which are expensive and laborious to obtain. In this work, we investigate the generation of synthetic training data for video action recognition, as synthetic data have been successfully used to supervise models for a variety of other computer vision tasks. We propose an interpretable parametric generative model of human action videos that relies on procedural generation, physics models and other components of modern game engines. With this model we generate a diverse, realistic, and physically plausible dataset of human action videos, called PHAV for "Procedural Human Action Videos". PHAV contains a total of 39,982 videos, with more than 1000 examples for each of 35 action categories. Our video generation approach is not limited to existing motion capture sequences: 14 of these 35 categories are procedurally-defined synthetic actions. In addition, each video is representedwith 6 different data modalities, including RGB, optical flow and pixel-level semantic labels. These modalities are generated almost simultaneously using the Multiple Render Targets feature of modern GPUs. In order to leverage PHAV, we introduce a deep multi-task (i.e. that considers action classes from multiple datasets) representation learning architecture that is able to simultaneously learn from synthetic and real video datasets, even when their action categories differ. Our experiments on the UCF-101 and HMDB-51 benchmarks suggest that combining our large set of synthetic videos with small real-world datasets can boost recognition performance. Our approach also significantly outperforms video representations produced by fine-tuning state-of-the-art unsupervised generative models of videos.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Semantic monocular depth estimation based on artificial intelligence

  • Gurram, Akhil|||0000-0002-2544-1510
  • Urfalioglu, Onay
  • Halfaoui, Ibrahim
  • Bouzaraa, Fahd
  • López Peña, Antonio M.|||0000-0002-6979-5783
Depth estimation provides essential information to perform autonomous driving and driver assistance. A promising line of work consists of introducing additional semantic information about the traffic scene when training CNNs for depth estimation. In practice, this means that the depth data used for CNN training is complemented with images having pixel-wise semantic labels where the same raw training data is associated with both types of ground truth, i.e., depth and semantic labels. The main contribution of this paper is to show that this hard constraint can be circumvented, i.e., that we can train CNNs for depth estimation by leveraging the depth and semantic information coming from heterogeneous datasets. In order to illustrate the benefits of our approach, we combine KITTI depth and Cityscapes semantic segmentation datasets, outperforming state-of-the-art results on monocular depth estimation.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

3D Perception With Slanted Stixels on GPU

  • Hernández Juárez, Daniel|||0000-0001-5878-1549
  • Espinosa, Antonio|||0000-0002-6460-3789
  • Vázquez Bermúdez, David|||0000-0002-2845-8158
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Moure, Juan C.|||0000-0001-6697-0331
This article presents a GPU-accelerated software design of the recently proposed model of Slanted Stixels, which represents the geometric and semantic information of a scene in a compact and accurate way. We reformulate the measurement depth model to reduce the computational complexity of the algorithm, relying on the confidence of the depth estimation and the identification of invalid values to handle outliers. The proposed massively parallel scheme and data layout for the irregular computation pattern that corresponds to a Dynamic Programming paradigm is described and carefully analyzed in performance terms. Performance is shown to scale gracefully on current generation embedded GPUs. We assess the proposed methods in terms of semantic and geometric accuracy as well as run-time performance on three publicly available benchmark datasets. Our approach achieves real-time performance with high accuracy for 2048 × 1024 image sizes and 4 × 4 Stixel resolution on the low-power embedded GPU of an NVIDIA Tegra Xavier.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Multimodal end-to-end autonomous driving

  • Xiao, Yi|||0000-0003-0486-7472
  • Codevilla Moraes, Felipe|||0000-0002-9815-6841
  • Gurram, Akhil|||0000-0002-2544-1510
  • Urfalioglu, Onay
  • López Peña, Antonio M.|||0000-0002-6979-5783
A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g. , LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities, i.e. using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Variable rate deep image compression with modulated autoencoder

  • Yang, Fei|||0000-0003-4099-6511
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • Iglesias-Guitian, Jose A.|||0000-0002-0817-1010
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Mozerov, Mikhail|||0000-0002-1117-743X
Variable rate is a requirement for flexible and adaptable image and video compression. However, deep image compression methods (DIC) are optimized for a single fixed rate-distortion (R-D) tradeoff. While this can be addressed by training multiple models for different tradeoffs, the memory requirements increase proportionally to the number of models. Scaling the bottleneck representation of a shared autoencoder can provide variable rate compression with a single shared autoencoder. However, the R-D performance using this simple mechanism degrades in low bitrates, and also shrinks the effective range of bitrates. To address these limitations, we formulate the problem of variable R-D optimization for DIC, and propose modulated autoencoders (MAEs), where the representations of a shared autoencoder are adapted to the specific R-D tradeoff via a modulation network. Jointly training this modulated autoencoder and the modulation network provides an effective way to navigate the R-D operational curve. Our experiments show that the proposed method can achieve almost the same R-D performance of independent models with significantly fewer parameters.
Dipòsit Digital de Documents de la UAB
Publication Artículo científico (JournalArticle)

Monocular depth estimation through virtual-world supervision and real-world SfM self-supervision

  • Gurram, Akhil|||0000-0002-2544-1510
  • Tuna, Ahmet Faruk|||0000-0003-3523-511X
  • Shen, Fengyi
  • Urfalioglu, Onay
  • López Peña, Antonio M.|||0000-0002-6979-5783
Depth information is essential for on-board perception in autonomous driving and driver assistance. Monocular depth estimation (MDE) is very appealing since it allows for appearance and depth being on direct pixelwise correspondence without further calibration. Best MDE models are based on Convolutional Neural Networks (CNNs) trained in a supervised manner, i.e., assuming pixelwise ground truth (GT). Usually, this GT is acquired at training time through a calibrated multi-modal suite of sensors. However, also using only a monocular system at training time is cheaper and more scalable. This is possible by relying on structure-from-motion (SfM) principles to generate self-supervision. Nevertheless, problems of camouflaged objects, visibility changes, static-camera intervals, textureless areas, and scale ambiguity, diminish the usefulness of such self-supervision. In this paper, we perform monocular d epth e stimation by v irtual-world s upervision (MonoDEVS) and real-world SfM self-supervision. We compensate the SfM self-supervision limitations by leveraging virtual-world images with accurate semantic and depth supervision, and addressing the virtual-to-real domain gap. Our MonoDEVSNet outperforms previous MDE CNNs trained on monocular and even stereo sequences.