The Engineering Unlocks Behind DeepSeek

DeepSeek V3 is a general-purpose model released in December 2024, while R1 is a reasoning model built on top of V3. The host says R1 reached performance comparable to OpenAI o1 and Google Flash 2.0 on some complex reasoning benchmarks. Many of the techniques behind the models appeared in DeepSeek papers published earlier in 2024.

V3 was trained in FP8, with periodic FP32 accumulation to limit numerical errors while using less memory. Its mixture-of-experts design has 671 billion parameters but activates 37 billion per token, and multi-head latent attention reduces KV cache storage. Multi-token prediction adds training signals and can also support faster generation.

DeepSeek used reinforcement learning on problems with verifiable answers, especially in math and coding, scoring final answers with simple accuracy and formatting rules. Its GRPO method updates the model from those scores without supplying examples of how to reason. Reasoning emerged over thousands of steps, but the initial R1-Zero outputs mixed languages and were hard to read, so DeepSeek added a structured reasoning fine-tuning phase.

The host notes that R1 performed comparably to o1 on some math and coding benchmarks, while OpenAI’s o3-mini soon outperformed both on key benchmarks. R1’s free access and lower cost helped draw attention, but the reported $5.5 million for V3 covered only its final training run, not R1 training, research, or hardware operating costs. The host says a Berkeley lab reproduced key R1-Zero techniques in a smaller model for $30.

The host argues that DeepSeek shows there is room for new companies to improve GPU workloads, inference software, tooling, and AI-generated kernels. Lower intelligence costs could benefit consumer and business applications. The episode closes with a YC Spring Batch application deadline of February 11.