Designing Characters with Deep Learning: Spellbrush - YC Gaming Tech Talks 2020

Corey says Spellbrush is building deep learning tools for artists because illustration takes time and art can consume 50 to 70 percent of a AAA game's production budget. He demonstrates that the company's model can generate an anime-style character in under two seconds, compared with the hours an illustrator might spend on one. Waifu Labs lets users customize characters through an online flow.

Spellbrush uses generative adversarial networks, pairing a generator that makes images with a discriminator that judges them against a corpus of real art. The two networks update based on the discriminator's errors over millions of training cycles. Random input, called latent noise, lets the generator produce varied images.

Corey describes using the generator to vary a character's expressions, colors, and illustration style. Spellbrush trains on roughly 10 million publicly available anime images, a dataset in which female characters outnumber male characters about six to one and darker skin tones appear in under 3 percent of illustrations. The team says it adjusts generation to include more male characters and darker skin tones.

Spellbrush built an office cluster with more than 200 CPU cores, over 20 GPUs, and extensive storage. Corey estimates comparable cloud training would cost $3,000 to $4,000 per model over seven to ten days, while the cluster costs about 60 cents per hour to run. The team uses its NetGen language, TensorFlow, Singularity, Slurm, Prometheus, Grafana, and TensorBoard in its training workflow.

Spellbrush is researching automated animation, workflow assistance for Live2D and Spine, and super-resolution for animation. The team says it is using its technology to build what it believes is the first AI-illustrated game. At the time of the talk, Spellbrush had five people and was hiring artists and an AI research intern.