Going In Deep On Data

François argues that production failures often come from gaps in training data, not model architecture. At Focal Systems, fogged refrigerator doors and people blocking products caused errors that adding layers could not fix. He says expert data and realistic RL environments are needed for fields such as medicine, law, and accounting.

Vincent Chen describes Snorkel’s work to turn expert judgment into reusable data. Its early data programming approach encoded labeling rules as software and combined noisy signals to estimate labels without ground truth. For more complex tasks, Snorkel’s SWE-bench Senior uses high-level user stories and validation agents that generate deterministic tests, alongside checks for correctness and code quality.

Volodymyr presents Inception Labs’ diffusion language models, which generate tokens in parallel refinement steps and, he says, can exceed 1,000 tokens per second. He describes Tau Forge, a system that uses real usage data or business information to synthesize environments and tasks, then adjusts task difficulty to produce useful learning signals. In an experiment, training on synthesized tasks improved Mercury 2.5’s accuracy on other environments by more than 23%.

Shane presents research on how languages help or interfere with one another during pretraining, using Thai as an example with just 0.6% as many tokens as English in MADLAD-400. The measured transfer is asymmetric: a language that helps Thai may not benefit another language in return, and script similarity mattered somewhat more than language family. The team’s scaling method combines target-language data with helpful transfer languages to estimate model performance and guide data mixtures.