- What We Do: We use LoRa models to efficiently adapt and create visual styles in our games, keeping the process quick and the file sizes small.
- Usefulness for Us: This technology speeds up our design process, enabling rapid experimentation and iteration.
- Benefits for Players: Players can use this technology to create their own concept art, adding a personal touch to their gaming experience.
“LoRa models” What are they, and how can they benefit your game design process? LoRA or Low-Rank Adaptation, applies small changes to the most critical part of Stable Diffusion models: The cross-attention layers, where the image and the prompt interact.
Unlike other training methods like Dreambooth, which are powerful but create huge files, or textual inversion, which makes tiny files but is less versatile, LoRA strikes a balance. It keeps file sizes manageable (2 – 200 MBs) while still offering solid training capabilities.
Here’s how it works
LoRA introduces small, trainable components into the base-model’s architecture — allowing for significant reductions in memory usage and computational demand. LoRA allows for greater experimentation and agility in 2d concept art. You can rapidly prototype new ideas and adapt traditional AI assets to better fit your game’s unique aesthetic and narrative style.
DreamBooth Implementation
We utilize DreamBooth to adapt a general diffusion model specifically to our artistic style, using a select number of images created by our SpaceEngineers Art Team. This technique helped us integrate our unique style onto three separate base models. For the purpose of training the models, we used only the art created in Keen Software House.
Testing Base Models
We work with our concept artists to quickly visualize sketches with the application of our base models, and assess concept viability. Although this method speeds up the early prototyping phase, making these concepts fully functional still requires some extra hands-on work.
To address this, we introduce customized LoRa models.
Creating LoRas
By adding LoRas, we enhance our base models with components that serve as an emphasis onto the original style, resulting in promising outputs with just a single generation. Like with our general diffusion models, these LoRas are trained on images exclusively sourced from our concept artists, ensuring clear understanding of the artistic and technical nuances of our game.
Empowering Artists
This combination of models allow our artists to experiment with and refine potential designs efficiently, setting a precise foundation early in the creative process.
Resources:
- Stable Diffusion: https://huggingface.co/stabilityai/stable-diffusion-2
- Automatic1111: https://github.com/AUTOMATIC1111/stable-diffusion-webui
- ControlNET: https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main
- Lora Models: