Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 3
Encontrada(s) 1 página(s)

Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Domain-Adaptive Deep Network Compression

  • Masana, Marc|||0000-0003-3254-3096
  • Weijer, Joost van de|||0000-0002-9656-9706
  • Herranz, Luis|||0000-0002-7022-3395
  • Bagdanov, Andrew|||0000-0001-6408-7043
  • Alvarez, Jose M.
Deep Neural Networks trained on large datasets can be easily transferred to new domains with far fewer labeled examples by a process called fine-tuning. This has the advantage that representations learned in the large source domain can be exploited on smaller target domains. However, networks designed to be optimal for the source task are often prohibitively large for the target task. In this work we address the compression of networks after domain transfer. We focus on compression algorithms based on low-rank matrix decomposition. Existing methods base compression solely on learned network weights and ignore the statistics of network activations. We show that domain transfer leads to large shifts in network activations and that it is desirable to take this into account when compressing. We demonstrate that considering activation statistics when compressing weights leads to a rank-constrained regression problem with a closed-form solution. Because our method takes into account the target domain, it can more optimally remove the redundancy in the weights. Experiments show that our Domain Adaptive Low Rank (DALR) method significantly outperforms existing low-rank compression techniques. With our approach, the fc6 layer of VGG19 can be compressed more than 4x more than using truncated SVD alone - with only a minor or no loss in accuracy. When applied to domain-transferred networks it allows for compression down to only 5-20% of the original number of parameters with only a minor drop in performance.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Transferring GANs, Generating images from limited data

  • Wang, Yaxing|||0000-0002-6055-7164
  • Wu, Chenshen
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • Gonzalez-Garcia, Abel
  • Raducanu, Bogdan|||0000-0003-3648-8020
Transferring knowledge of pre-trained networks to new domains by means of fine-tuning is a widely used practice for applications based on discriminative models. To the best of our knowledge this practice has not been studied within the context of generative deep networks. Therefore, we study domain adaptation applied to image generation with generative adversarial networks. We evaluate several aspects of domain adaptation, including the impact of target domain size, the relative distance between source and target domain, and the initialization of conditional GANs. Our results show that using knowledge from pre-trained networks can shorten the convergence time and can significantly improve the quality of the generated images, especially when target data is limited. We show that these conclusions can also be drawn for conditional GANs even when the pre-trained model was trained without conditioning. Our results also suggest that density is more important than diversity and a dataset with one or few densely sampled classes is a better source model than more diverse datasets such as ImageNet or Places.
Dipòsit Digital de Documents de la UAB
Publication Capítulo o parte de un libro (BookPart)

Rotate your networks, better weight consolidation and less catastrophic forgetting

  • Liu, Xialei
  • Masana, Marc|||0000-0003-3254-3096
  • Herranz, Luis|||0000-0002-7022-3395
  • Weijer, Joost van de|||0000-0002-9656-9706
  • López Peña, Antonio M.|||0000-0002-6979-5783
  • Bagdanov, Andrew|||0000-0001-6408-7043
In this paper we propose an approach to avoiding catastrophic forgetting in sequential task learning scenarios. Our technique is based on a network reparameterization that approximately diagonalizes the Fisher Information Matrix of the network parameters. This reparameterization takes the form of a factorized rotation of parameter space which, when used in conjunction with Elastic Weight Consolidation (which assumes a diagonal Fisher Information Matrix), leads to significantly better performance on lifelong learning of sequential tasks. Experimental results on the MNIST, CIFAR-100, CUB-200 and Stanford-40 datasets demonstrate that we significantly improve the results of standard elastic weight consolidation, and that we obtain competitive results when compared to the state-of-the-art in lifelong learning without forgetting.