The group reviews Sora clips, noting clear text, detailed images, prompt adherence, and consistency across a minute-long scene. They also point out flaws: a floating object, distorted bridge columns, cars on the wrong side of the road, and unnatural waves.
How To Build Generative AI Models Like OpenAI's Sora
Y Combinator hosts and guests discuss Sora’s video generation and how startups can build specialized AI models with limited resources on The Light Cone podcast.
A guest describes Sora as combining Transformers and diffusion, with spatiotemporal patches representing image areas across multiple frames. They connect the approach to earlier work on Vision Transformers and world models, while stressing that details of Sora’s design are not public and some explanations are speculation.
The hosts describe YC companies using Azure GPU credits and targeted methods to train or adapt models during a batch. Examples include Infinity AI adapting a model with about an hour of video, SyncLabs training lip-sync models on one A100, and Sonado’s 21-year-old founders building a text-to-song model.
Metalware used scanned hardware textbooks and a smaller model for its design copilot; Find generated synthetic programming competition data for software answers. The discussion also covers models for weather, proteins, EEG signals, robotics, and CAD, where domain expertise or tailored data and computation can make narrow applications practical.
Playground is presented as an example of a company that pivoted into AI, with founder Suhail Doshi studying papers before building image models. The hosts argue that startups can compete in valuable specialized applications without matching the funding or scale of large foundation-model companies.
Takeaways
- Choose a narrow task where specialized data can outperform a larger general-purpose model.
- High-quality, task-specific data can reduce the compute needed to train a useful model.
- Use pretrained models and small datasets to adapt outputs to a specific person or workflow.
- GPU credits and faster iteration can help small teams test model ideas during a short build cycle.
- Domain knowledge can reveal useful model applications beyond text and image generation.
Mentioned in this talk
Search inside this talk
What was claimed 10
- built
“We got access to Sora”
YouTube Verified2 sources· Talk, Web· archived
- feature
Sora: Long-term visual consistency across a minute-long video“they have long-term visual consistency”
YouTube Unverified1 source· Talk
- uses
“train it with videos and what they call spatiotemporal patches”
YouTube Unverified1 source· Talk
- usesSora uses diffusion models
“and a diffusion model”
YouTube Unverified1 source· Talk
- usesSora uses Transformer++
“it’s a combination of a Transformer model”
YouTube Unverified1 source· Talk
- metricSyncLabs: Trained on a single A100
“they trained the models on a single A100”
YouTube Unverified1 source· Talk
- usesFind uses synthetic dataprogramming competition datasets
“They created a bunch of synthetic data for programming competitions”
YouTube Unverified1 source· Talk
- metricPyramidal: 800 hours of GPU compute for a single iteration of an initial model
“a single iteration of an initial model with just 800 hours of GPU compute”
YouTube Unverified1 source· Talk
