Lessons From Training Composer At Cursor And Building Meta/Nvidia Compute Clusters

Stuart So introduces ParallelKittens, a CUDA framework for making multi-GPU kernels easier to write and maintain. He says networking can consume up to 50% of runtime in some workloads and explains that performance depends on choosing the right data transfer and scheduling strategy. He reports that kernels written with about 50 to 100 lines of device code can match or beat hand-optimized kernels with hundreds or thousands of lines; Cursor uses the framework to train Composer on tens of thousands of Blackwell GPUs.

John presents a study of more than 20 local models, from 1 to 200 billion parameters, across accelerators from Apple, Nvidia, AMD, and SambaNova. He says up to 88.7% of queries could be routed to local open-source models with a perfect router, and reports a roughly threefold gain in intelligence per watt over two years and an 18-fold gain in intelligence per joule over about 16 months. He argues that local and cloud inference can share workloads, with even imperfect routing potentially reducing energy, compute, and dollar costs by 50% to 70%.

Mark describes KernelBot, a GPU kernel competition platform, and KernelGuard, which detects benchmark cheating. He gives examples of reward hacks, including a vector-mean kernel that returns zero for inputs sampled around a mean of zero and a kernel that behaves correctly during checks but incorrectly during performance tests. A community search for QR decomposition kernels produced a kernel reported to be 60 times faster than the baseline, while exposing an open challenge: turning long, shape-specific submissions into maintainable code.

Misha Manski of Marlo explains that inference phases stress compute, memory, and networks differently: prefill tends to be compute-bound, while decode is often memory-bandwidth-bound and latency-sensitive. He outlines ways to split work across systems, such as placing decode on an SRAM-based accelerator or separating attention from the MLP. Whether these designs pay off depends on output length, concurrency, latency needs, and the added power and networking costs.

Brennan presents Stanford research on using entity-component-system patterns to run batches of game environments on a GPU for reinforcement learning. The approach stores environment data in GPU memory and runs systems over entities in parallel, supporting irregular gameplay logic and dynamic allocation. On an RTX 4090, the team reports more than 100 times the throughput of CPU reference implementations for several tested environments, including hide-and-seek and Overcooked.