Building The World's Best Image Diffusion Model

Doshi describes Playground as a visual-first tool: users can start with a template or upload an image, then ask for specific changes in plain English. The examples include adding a two-fan GPU to a shirt design, changing a background, and positioning accurate text. He says the team prioritized text because it makes generated graphics useful for things like logos, posters, and shirts.

After observing users struggle to prompt and repeatedly regenerate images, the team built around templates that people could modify. Doshi says users should not have to become prompt engineers; Playground should handle the prompting work behind the scenes. The team shifted its focus toward graphics and design after deciding that its earlier audience and use cases were not the business it wanted to build.

Doshi says Playground v3 uses a model architecture built from scratch because the team felt CLIP limited prompt understanding and an off-the-shelf VAE could not preserve small details. The team made a risky architecture choice despite uncertainty about whether it would work. It also trained on detailed captions to improve spatial relationships, emotional expressions, text, and other prompt details.

Playground runs user evaluations, including comparisons between its model and competitors. Doshi says close prompt adherence can lower aesthetic ratings when users prefer a more pleasing image that ignores part of the prompt. He argues that this creates an evaluation problem: measuring aesthetics separately from instruction-following is difficult.

Doshi connects Playground’s shift in focus to his earlier experience at Mixpanel, where he chose a broader market over churn-prone gaming customers, and at Mighty, where he stopped pursuing a browser company after hitting technical and market headwinds. He says Playground benefits from AI tailwinds, but building a strong model still requires close attention to details such as kerning, skin texture, and film grain. He describes the work as repeated scrutiny and debate across the research team.