Skip to main content

Resultados relacionados

Resultados totales (Incluyendo duplicados): 3
Encontrada(s) 1 página(s)

Academica-e. Repositorio Institucional de la Universidad Pública de Navarra
Publication Contribución de congreso (ConferenceOutput)

Enhancing DreamBooth with LoRA for generating unlimited characters with stable diffusion

  • Pascual Casas, Rubén
  • Maiza Coupin, Adrián Mikel
  • Sesma Sara, Mikel
  • Paternain Dallo, Daniel
  • Galar Idoate, Mikel
This paper addresses the challenge of generating unlimited new and distinct characters that encompass the style and shared visual characteristics of a limited set of human designed characters. This is a relevant problem in the audiovisual industry, as the ability to rapidly produce original characters that adhere to specific characteristics greatly increases the possibilities in the production of movies, series, or video games. Our solution is built upon DreamBooth, a widely extended fine-tuning method for text-to-image models. We propose an adaptation focusing on two main challenges: the impracticality of relying on detailed image prompts for character description and the few-shot learning scenario with a limited set of characters available for training. To solve these issues, we introduce additional character-specific tokens to DreamBooth training and remove its class-specific regularization dataset. For an unlimited generation of characters, we propose the usage of random tokens and random embeddings. This proposal is tested on two specialized datasets and the results shows our method¿s capability to produce diverse characters that adhere to a style and visual characteristics. An ablation study to analyze the contributions of the proposed modifications is also developed., This work has been funded by MCIN/AEI/10.13039/501100011033/FEDER, UE, with the project PID2022-136627NB-I00, by the Government of Navarre under the project 0011-1365-2022-000130, and by the Public University of Navarra under the project PJUPNA2023-11377. The Scary and Virus datasets are designed by Freepik. This research received support from an FPU grant (Formación de Profesorado Universitario) awarded by the Spanish Ministry of Science and Innovation (MCINN) to Rubén Pascual.
Academica-e. Repositorio Institucional de la Universidad Pública de Navarra
Publication Contribución de congreso (ConferenceOutput)

Generación ilimitada de personajes mediante Stable Diffusion con DreamBooth y LoRA

  • Pascual Casas, Rubén
  • Maiza Coupin, Adrián Mikel
  • Sesma Sara, Mikel
  • Paternain Dallo, Daniel
  • Galar Idoate, Mikel
Este artículo aborda el reto de generar un número ilimitado de personajes nuevos, y distintos, que engloben el estilo y las características visuales compartidas de un conjunto limitado de personajes diseñados por un humano. Este es un problema de gran relevancia en la industria audiovisual, ya que la capacidad de producir rápidamente personajes originales que se adhieran a unas características específicas aumenta enormemente las posibilidades en la producción de películas, series o videojuegos. Nuestra solución se basa en DreamBooth, un método de ajuste de modelos generativos de texto a imagen ampliamente extendido. Proponemos una adaptación centrada en dos retos principales: lo poco práctico que resulta utilizar prompts detallados de las imágenes para describir los personajes y la complejidad del ajuste de modelos a partir de un conjunto limitado de personajes. Para resolver estos problemas, introducimos en el entrenamiento de DreamBooth tokens adicionales específicos para cada personaje y eliminamos el conjunto de datos de regularización. Para generar personajes de manera ilimitada, proponemos el uso de tokens y embeddings aleatorios. Comprobamos la utilidad de la propuesta utilizando dos conjuntos de datos diferentes. Los resultados obtenidos muestran la capacidad de nuestro método para producir personajes diversos que se adhieren a un estilo y a unas características visuales concretas. Finalmente, desarrollamos un estudio de ablación., Este trabajo está financiado por MCIN/AEI/10.13039/501100011033/FEDER, UE, con el proyecto PID2022-136627NB-I00, por el Gobierno de Navarra bajo el proyecto 0011-1365-2022-000130, y por la Universidad Pública de Navarra bajo el proyecto PJUPNA2023-11377. Los conjuntos de datos Scary y Virus están diseñados por Freepik. Esta investigación ha contado con el apoyo de una beca FPU concedida por el Ministerio de Ciencia e Innovación de España (MCINN) a Rubén Pascual.
Academica-e. Repositorio Institucional de la Universidad Pública de Navarra
Publication Artículo científico (JournalArticle)

Few-shot multi-token DreamBooth with LoRa for style-consistent character generation

  • Pascual Casas, Rubén
  • Sesma Sara, Mikel
  • Jurío Munárriz, Aránzazu
  • Paternain Dallo, Daniel
  • Galar Idoate, Mikel
The audiovisual industry is undergoing a profound transformation as it is integrating AI developments not only to automate routine tasks but also to inspire new forms of art. This paper addresses the problem of producing a virtually unlimited number of novel characters that preserve the artistic style and shared visual traits of a small set of human-designed reference characters, thus broadening creative possibilities in animation, gaming, and related domains. Our solution builds upon DreamBooth, a well-established fine-tuning technique for text-to-image diffusion models, and adapts it to tackle two core challenges: capturing intricate character details beyond textual prompts and the few-shot nature of the training data. To achieve this, we propose a multi-token strategy, using clustering to assign separate tokens to individual characters and their collective style, combined with LoRA-based parameter-efficient fine-tuning. By removing the class-specific regularization set and introducing random tokens and embeddings during generation, our approach allows for unlimited character creation while preserving the learned style. We evaluate our method on five small specialized datasets, comparing it to relevant baselines using both quantitative metrics and a human evaluation study. Our results demonstrate that our approach produces high-quality, diverse characters while preserving the distinctive aesthetic features of the reference characters, with human evaluation further reinforcing its effectiveness and highlighting the potential of our method., Rubén Pascual received support from an FPU grant (Formación de Profesorado Universitario, ref. FPU22/02961) awarded by the Spanish Ministry of Science and Innovation (MCINN). This work was also partially supported by the Spanish Ministry of Science and Innovation under project PID2022-136627NB-I00 (MCIN/AEI/10.13039/ 501100011033/FEDER, UE) and by the Gobierno de Navarra under projects INVERSO (0011-1365-2022-000130) and Genesis (PC24- GENESIS-041-002).