DeepSeek R1: Open-Source Reasoning Model
DeepSeek releases R1 reasoning model, matching o1 performance and fully open-sourced.
Overview
In January 2025, DeepSeek released R1 reasoning model. R1 is a reasoning model comparable to OpenAI o1, performing excellently on math, coding, and reasoning tasks. More importantly, DeepSeek fully open-sourced R1 model weights and technical report, detailing how to train reasoning capabilities through reinforcement learning. R1's release shocked the AI community, proving open-source models could achieve top-tier performance in reasoning. It significantly lowered the barrier to deploying high-performance reasoning models, advancing AI technology democratization.
Study Notes
Background, mechanics, impact, and key takeaways in one read
Background & Context
DeepSeek-R1 was released by DeepSeek on January 20, 2025 in the paper "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (arXiv:2501.12948), open-weight under the MIT license. The backdrop: OpenAI o1 had shown "test-time compute" works but stayed closed. DeepSeek's aim was to reproduce long-chain reasoning and self-verification with pure reinforcement learning——relying little on human chain-of-thought labels——on open weights, and to prove it can be distilled to small models. R1 builds on DeepSeek-V3 (a 671B MoE with about 37B active per token) and matches o1 on AIME, MATH-500, and Codeforces, a milestone for open reasoning models. The lab, founded in 2023 and backed by the quant fund High-Flyer, had earlier shipped the cost-efficient DeepSeek-V3; R1 adds reasoning training on that base and opened an API at roughly one-thirtieth of o1's price, drawing global attention.
Key Points
- 1Built on DeepSeek-V3 (671B MoE, ~37B active per token) and fully open-weight under the MIT license.
- 2R1-Zero skips SFT and uses pure GRPO, spontaneously showing an "aha moment," long CoT, and self-verification.
- 3R1 adds a cold-start SFT and multi-stage RL to fix R1-Zero's readability and language-mixing problems.
- 4GRPO replaces PPO: it uses the group mean and variance as the baseline, dropping a separate value network and saving memory.
- 5It scores 79.8% on AIME 2024, 97.3% on MATH-500, 71.5% on GPQA diamond, matching o1.
- 6Reasoning behavior is distilled into Qwen/Llama dense models from 1.5B to 70B, giving small models strong reasoning.
How It Works
R1 has two routes. The extreme R1-Zero applies pure GRPO directly on the V3 base: rewards are only verifiable rules——math by final answer, code by test execution——plus a format reward for wrapping reasoning in tags; no neural reward model is trained, to avoid reward hacking. Without any human chain-of-thought demonstration, the model spontaneously learns reflection and backtracking; the paper records it writing "wait, that's an aha moment," yet it suffers language mixing and messy formatting. R1 therefore cold-starts: a few thousand curated long chains do SFT to stabilize format, then large-scale GRPO, then rejection sampling builds about 600k reasoning samples for a second SFT, and finally a general RL mixes rule-based and preference rewards. GRPO's trick is dropping PPO's critic network and using the relative quality of several samples in a group as the baseline, saving memory.
Impact & Legacy
With fully open weights and reproducible details, DeepSeek-R1 proved top reasoning need not sit behind a closed API, triggering global follow-up on open reasoning models (e.g., thinking modes in Qwen and Llama). It gave the first public evidence that pure RL can elicit reasoning and established GRPO and small-model distillation as practical routes. Limits include high latency and cost from long chains, filtering of sensitive content, and weaker multimodal and safety evaluation; its low-cost high performance also spurred debate on training efficiency and compute control.
Key Takeaways
- Pure RL (GRPO plus verifiable rewards) can elicit long-chain reasoning from scratch, without first imitating humans.
- Cold-start SFT is a pragmatic trade-off between readability and pure-RL exploration.
- GRPO drops the critic and uses in-group relative comparison as baseline, memory-friendly for small-model RL.
- Distilling a large model's reasoning into small ones brings strong reasoning to consumer hardware.
Glossary
A reinforcement learning algorithm without a value network, using the relative quality of a group's samples as baseline to optimize policy, saving memory.
A supervised fine-tuning initialization step before large-scale RL, using a few high-quality demonstrations to stabilize format and readability.
Training a small model (student) on a large model's (teacher's) outputs to inherit its ability; R1 distills reasoning into 1.5B–70B dense models.