Academica-e. Repositorio Institucional de la Universidad Pública de Navarra
Publicación
Contribución de congreso (ConferenceOutput)
Enhancing DreamBooth with LoRA for generating unlimited characters with stable diffusion
This paper addresses the challenge of generating unlimited new and distinct characters that encompass the style and shared visual characteristics of a limited set of human designed characters. This is a relevant problem in the audiovisual industry, as the ability to rapidly produce original characters that adhere to specific characteristics greatly increases the possibilities in the production of movies, series, or video games. Our solution is built
upon DreamBooth, a widely extended fine-tuning method for text-to-image models. We propose an adaptation focusing on two main challenges: the impracticality of relying on detailed image prompts for character description and the few-shot learning scenario with a limited set of characters available for training. To solve these issues, we introduce additional character-specific tokens to DreamBooth training and remove its class-specific regularization dataset. For an unlimited generation of characters, we propose the usage of random tokens and random embeddings. This proposal is tested on two specialized datasets and the results shows our method¿s capability to produce diverse characters that adhere to a style and visual characteristics. An ablation study to analyze the contributions of the proposed modifications is also developed., This work has been funded by MCIN/AEI/10.13039/501100011033/FEDER, UE, with the project PID2022-136627NB-I00, by the Government of Navarre under the project 0011-1365-2022-000130, and by the Public University of Navarra under the project PJUPNA2023-11377. The Scary and Virus datasets are designed by Freepik. This research received support from an FPU grant (Formación de Profesorado Universitario) awarded by the Spanish Ministry of Science and Innovation (MCINN) to Rubén Pascual.