Reflection Beam 1.0: 501B open MoE model with 1M context, Apache 2.0
Reflection's Beam-1.0-501B-A23B is a 501B-parameter open mixture-of-experts model with 23B active parameters per token, a 1M-token context window, and an Apache 2.0 license. It is the first model from the new US lab Reflection, and it matters because it targets coding and agentic workloads at frontier scale while being deployable and fine-tunable on hardware teams already own.
Beam is text-in, text-out only. It ships with a dedicated open-source Rust-based harness, native transformers support, and day-0 integration with vLLM and SGLang. Hugging Face and Reflection also published inference, supervised fine-tuning (SFT), and reinforcement learning (RL) recipes for the model.
Key specifications at a glance
Beam is a sparse MoE model that runs at the speed of a much smaller model while retaining the capacity of a large one, because each token is computed by only a fraction of the network.
- 501B total parameters, 23B active per token
- 52 layers, 256 experts per MoE layer, 8 routed per token
- Text in, text out, with a 1M-token context window
- Checkpoint formats: BF16, FP8, and NVFP4
- Reasoning effort values: "low", "medium" (default), "high", "xhigh", and "max"
- Reasoning that can stay in context across turns via
preserve_thinking - Tool calling with a dedicated grammar
- Response template stored in the tokenizer
- Speculative decoding drafter model with DFlash2
- Day-0 libraries: transformers, SGLang, vLLM; full-parameter RL with Miles
- Apache 2.0 license
Architecture: hybrid attention, grouped experts, and a constant-scale residual stream
Beam's decoder has 52 layers with a hidden size of 5,120. The first 3 layers use a dense SwiGLU MLP; the other 49 replace it with a Mixture-of-Experts block.
Hybrid attention: 43 sliding-window layers and 9 global layers
Most layers use sliding-window attention over the previous 512 tokens. Every sixth layer, plus the last one, attend to the full context. That produces 43 sliding-window layers and 9 global layers.
The two layer types handle position differently. Sliding-window layers use RoPE, because relative position matters most within a short window. Global layers use no positional encoding at all. Instead, each query is multiplied by a scaler that grows slowly with the number of visible tokens, (1 + (pos + 1) / 2048) ** 0.1. The implementation calls this scalable-softmax query scaling, and it acts as a temperature so the softmax keeps its precision as context lengthens instead of flattening out.
Attention uses grouped-query attention with 48 query heads and 8 key-value heads, with RMSNorm applied to queries and keys. A learned sigmoid gate, computed from the layer input, scales every channel of every head before the output projection.
Mixture of Experts: 256 experts, 8 routed per token
Each of the 49 MoE layers contains 256 experts, and each expert is a SwiGLU MLP with an intermediate size of 2,560, or about 39M parameters. The experts hold 98.4% of the model's parameters.
The router follows DeepSeek-V3's group-limited scheme. It scores all 256 experts with a sigmoid, splits them into 8 groups of 32, keeps the 4 best groups ranked by the sum of each group's top two scores, then picks the top 8 experts inside them. Beam uses the same group sizes as DeepSeek-V3 but has no shared expert. Expert outputs are combined with normalized router weights and scaled by 2.5.
Because only 8 experts run per token, the model computes each token with about 23B parameters. All 256 experts stay in memory, so deployment needs the memory of a 501B model but runs at the speed of a 23B one. The BF16 checkpoint is about 1 TB in safetensors format. The NVFP4 checkpoint stores most parameters in 4-bit precision for NVIDIA Blackwell GPUs, with some parameters kept in BF16 for accuracy. FP8 checkpoints are also on the Hub and are described as ideal for Hopper and Ampere series chips.
Residual stream: depth-dependent weights keep the hidden state at constant scale
Beam's residual connections are its most distinctive architectural feature. A standard transformer adds each sublayer output to the stream with weight 1, so the stream grows with depth.
Beam counts the input embedding as step 1. At step k it adds the sublayer output with weight 1/sqrt(k) and keeps the existing stream with weight sqrt((k-1)/k). The squares of the two weights sum to one, so the stream stays at a constant scale from the first layer to the last. After all 104 additions, every sublayer output and the input embedding carry the same weight in the final hidden state.
Every sublayer is also normalized before and after, and the final logits pass through a tanh softcap at ±30, computed in FP32. The implementation fixes where it computes in FP32: the router, the SwiGLU product, the norms, RoPE, the query scaling, and the softcap.
Long context: the KV cache grows at roughly one sixth the naive rate
Hybrid attention is what makes Beam's 1M-token context practical. Each layer caches 2 x 8 KV heads x 128 dims = 2,048 values per token. If all 52 layers kept every token, the cache would grow by 208 KiB per token. Only the 9 global layers keep the whole sequence; the 43 sliding-window layers hold a fixed 512-token window each and stop growing. The cache therefore grows by 36 KiB per token, roughly a sixth of the naive rate.
| Context | If all 52 layers were global | Beam |
|---|---|---|
| 32K tokens | 6.5 GiB | 1.2 GiB |
| 128K tokens | 26 GiB | 4.6 GiB |
| Full context (1M) | 208 GiB | 36.08 GiB |
These numbers are per sequence, in BF16, with the transformers dynamic cache.
Benchmark results across BF16, FP8, and NVFP4
The following results are provided by the Reflection team. Quantization costs little accuracy across the three checkpoint formats.
| Benchmark | Beam (BF16) | Beam (FP8) | Beam (NVFP4) |
|---|---|---|---|
| Agentic Coding / Terminal | |||
| DeepSWE v1.1 | 43.3% | 43.0% | 42.7% |
| SWE Bench Pro v2-Hard | 76.5% | 78.4% | 78.6% |
| SWE Bench Pro v1 | 66.1% | 65.9% | 65.6% |
| Terminal Bench v2.1 | 80.1% | 78.9% | 78.2% |
| SWE Atlas Codebase QnA | 31.8% | 32.2% | 31.4% |
| SWEBench Multilingual | 78.3% | 78.1% | 77.5% |
| SWEBench Verified | 81.6% | 80.2% | 80.7% |
| Reasoning | |||
| AIME 2026 | 97.7% | 98.0% | 97.5% |
| HLE (no tools) | 36.0% | 36.0% | 34.8% |
| HLE (with tools) | 51.1% | 51.1% | 51.1% |
| SciCode | 49.2% | 48.3% | 46.9% |
| CriPT AA | 16.3% | 17.4% | 15.4% |
| GPQA Diamond | 90.6% | 90.3% | 89.3% |
| LiveCodeBench | 88.3% | 88.1% | 87.4% |
| Tool Calling / Search | |||
| AutomationBench (public split only) | 39.3 | 38.2 | 39.8% |
| MCP Atlas | 80.8% | 81.5% | 80.5% |
| Tau3 Banking | 36.1% | 36.8% | 35.4% |
| BrowseComp w/ context management | 76.8% | 76.7% | 75.6% |
| DeepSearchQA w/ context management | 80.5% | 79.6% | 81.1% |
| General Capabilities | |||
| AA-LCR | 78.2% | 79.4% | 78.7% |
| LongBench v2 | 67.5% | 67.6% | 67.2% |
| IFBench | 79.6% | 80.7% | 79.2% |
| AA Omniscience (index, public split only) | 10.8 | 8.8 | 9.8 |
Hardware requirements: NVFP4 is the most practical serving path
The most practical way to serve Beam is the NVFP4 checkpoint, which shrinks the 501B parameters from 1 TB in BF16 to roughly 280 GB while retaining most of the BF16 checkpoint's quality. Blackwell GPUs run NVFP4 natively, but GPUs older than Blackwell, such as the H200, cannot. FP8 checkpoints are the option for pre-Blackwell hardware.
A B200 has 180 GB of HBM3e, so two of them (360 GB) can hold the Beam weights with little room left for activations and the KV cache. The guidance is to plan for four B200s (720 GB), which leaves room for long contexts and concurrent requests. Beam can also be served on a full 8xB200 node, on two B300s at 288 GB each, or on 8xRTX Pro 6000. Teams without Blackwell hardware can test the model through Hugging Face Inference Providers.
Inference support: transformers, vLLM, and SGLang on day 0
Beam has day-0 support in transformers, SGLang, and vLLM.
Transformers
Install the latest transformers release:
pip install -U "transformers>=5.20.1"
The easiest entry point is the text-generation pipeline, which returns the whole chat with Beam's reply already parsed by the response template. For lower-level control, use AutoModelForCausalLM and AutoTokenizer.
Beam's tokenizer carries a response template. parse_response reads it and turns raw output into a message dict with the reasoning, the answer, and any tool calls in separate fields, avoiding the usual regex for splitting reasoning from answers from tool calls. Because the rules ship with the tokenizer, the parsed structure matches what Beam was trained to produce. The chat template can pre-write part of the assistant turn, which the parser needs to see.
For tool calling, pass tools to parse_response as well. Beam writes tool-call arguments inline in its own XML-style grammar, so a bare 7 could be a number or a string; with the tool schemas, the parser casts each argument to its declared type.
Multi-GPU and multi-node in transformers
Beam's config defines an expert-parallel plan. Expert parallelism puts each expert on one device, while attention, the dense layers, the embeddings, and the LM head are replicated on every device — that is 8B parameters, or 16 GB in BF16. Loading uses a DistributedConfig with tp_size and ep_size set to the world size. Launching with torchrun across 16 H200 GPUs means each holds 78 GB of weights; with 8 GPUs each would hold 139 GB, which leaves no room on a 141 GB H200. The number of devices must divide the number of experts, 256.
vLLM
Beam has day-0 support in vLLM. This command serves the BF16 checkpoint with tensor parallelism across 16 GPUs (two 8x H200 nodes) and exposes an OpenAI-compatible API:
vllm serve reflection/Beam-1.0-501B-A23B --tensor-parallel-size 16 --served-model-name beam
For faster generation, add DFlash2 speculative decoding. A small drafter model proposes up to 15 tokens ahead, and Beam checks them all in one forward pass, so each step can accept several tokens. The cost is a little extra memory for the drafter:
vllm serve reflection/Beam-1.0-501B-A23B \
--tensor-parallel-size 16 \
--speculative-config '{ "method": "dflash", "model": "reflection/dflash2-drafter-preview", "num_speculative_tokens": 15 }'
Both commands start an OpenAI-compatible server, so any OpenAI client can point at http://localhost:8000/v1.
SGLang
Beam also has day-0 support in SGLang, and the recommended configuration turns on DFlash2 speculative decoding. On Blackwell, the NVFP4 checkpoint fits on 4 B300s:
python -m sglang.launch_server \
--model-path reflection/Beam-1.0-501B-A23B-NVFP4 \
--served-model-name beam \
--tp 4 \
--attention-backend trtllm_mha \
--page-size 64 \
--context-length 1048576 \
--mem-fraction-static 0.85 \
--enable-unified-memory \
--speculative-algorithm DFLASH \
--speculative-draft-model-path reflection/Beam-1.0-501B-A23B-DFlash2 \
--speculative-dflash-block-size 8 \
--tool-call-parser beam --reasoning-parser beam \
--host 0.0.0.0 --port 30000
On Hopper GPUs, use the FP8 checkpoint on 8 H200s instead. Only three flags change: the checkpoint, the tensor-parallel size (8 instead of 4), and the attention backend (fa3 instead of trtllm_mha, which needs Blackwell).
Inference Providers and Inference Endpoints
Beam is available on Hugging Face Inference Providers, powered by Baseten, Fireworks, and Together, with more to follow. It is reachable through the OpenAI-compatible router at https://router.huggingface.co/v1 using the model ID reflection/Beam-1.0-501B-A23B:auto.
Hugging Face also set up a Beam recipe for one-click deployment in Inference Endpoints from the catalog in FP8 or NVFP4, or via the hf CLI:
hf endpoints catalog deploy --repo reflection/Beam-1.0-501B-A23B --name beam
Vibe evals: five tasks at low and high reasoning effort
Hugging Face ran what it calls vibe evals — not benchmarks — on five tasks, once each at low and high reasoning effort through the Reflection platform. The complete inputs, outputs, API usage, task fixtures, graders, and conversation traces are in a public Hugging Face bucket.
| Category | Prompt description | Low reasoning (tokens) | High reasoning (tokens) |
|---|---|---|---|
| Multi-step tool calling | Fulfill ORD-742: compare stock, reservations, surcharges and arrival dates; reserve the cheapest eligible warehouse. | Pass, 544 | Pass, 811 |
| Long-context retrieval over a code base | Derive Click command configuration and two invocation results from three complete source files; cite six implementation symbols. | Fail, 1,613 | Pass, 5,007 |
| Terminal task in a sandbox | Inspect and repair a CSV ledger CLI; fix all bugs, preserve tests, run tests and a CLI check. | Pass, 582 | Pass, 1,548 |
| Math reasoning (AIME-style) | How many eight-element subsets of {1, …, 16} have a sum divisible by 17? Give a justification. | Partial, 829 | Pass, 2,809 |
| Structured output | Convert an out-of-order revision log into strict JSON: merge patches, cancellations, nulls, dates, currencies and totals. | Pass, 1,040 | Pass, 2,088 |
Two caveats are documented in the source. On the low-effort long-context task, the response got all code behavior and all six source citations correct but did not correspond to the parser's format, which the authors attribute partly to prompt ambiguity. On the low-effort math task, the model returned the correct answer but omitted the requested justification; the high-effort run supplied a correct roots-of-unity proof that was independently reviewed.
Post-training: SFT with TRL and GRPO with Miles
Beam can be fine-tuned for custom use cases, and Hugging Face published examples in TRL and Miles for supervised fine-tuning and reinforcement learning.
SFT with TRL
The quickest way to adapt Beam to your own data is supervised fine-tuning with TRL's SFTTrainer. To keep memory manageable, the published example trains LoRA adapters with PEFT rather than updating all 501B parameters. The goal was to keep Beam's reasoning in the language requested by the user, fine-tuning on the Multilingual-Thinking dataset with LoRA, which trains a small set of adapter weights while keeping the base model frozen.
The setup combines model sharding at load time, expert parallelism, FlashAttention 4, and sequence packing. LoRA SFT fits on a single node with 8 B300 GPUs, reaching around 19,000 tokens per second; the corresponding H100 setup uses four nodes.
After one epoch of Multilingual-Thinking, Beam reasoned in the requested language on 16 out of 16 held-out prompts, compared with 3 out of 16 for the base model. The source notes this small evaluation measures language adherence, not the correctness of the reasoning itself.
GRPO with Miles
Reflection and RadixArk built a recipe that trains the model with GRPO in Miles, in BF16. It can update the full model or use LoRA to train adapters while keeping the base weights frozen. Megatron trains, SGLang generates the rollouts, and Miles publishes the new weights to the rollout engines after each update.
| Workload | Reference configuration |
|---|---|
| Inference / eval, BF16 | 16x H200 per engine |
| Full-parameter GRPO, BF16 (Miles) | 96x H200: 64 train, 32 rollout. Or 64x H200 colocated. |
| LoRA GRPO, BF16 (Miles) | 32x H200: 16 train, 16 rollout. |
| Full-parameter GRPO, BF16, 128-token context (Miles) | 48x H200: 32 train, 16 rollout. The setup Reflection and RadixArk first validated on the real checkpoint. |
The recipe runs GRPO on DAPO-Math-17K and evaluates on AIME25. A rollout works as follows:
- Sample: 32 prompts are passed to the model with 8 responses each. Two SGLang engines serve the rollouts on 16 GPUs each.
- Score: the boxed answer is checked against the label, with a soft penalty for responses near the length limit.
- Update: one optimizer step on all 256 samples, using group-relative advantages with truncated importance sampling and no KL or entropy penalty.
- Train in parallel: tensor parallelism 8, expert parallelism 8, data parallelism 8, with sequence parallelism and full recompute.
- Optimizer: Adam at a constant 1e-6. Gradients in BF16, master weights in FP32, moments in BF16. The optimizer state streams through node-local NVMe, so each trainer node needs 2 TB of scratch.
- Evaluate: AIME25 every five rollouts, eight samples per problem. A checkpoint is saved every twenty.
The next rollout is generated while the current one trains, so responses are one update behind. Adding --resume continues from the last checkpoint, and execute-colocate runs the same recipe on eight nodes with every GPU training and serving in turn.
Full-parameter GRPO raised AIME 2025 accuracy from 46.25% to 55.8% with a 4,000-token output limit. LoRA GRPO, which trains rank-16 adapters on Beam's attention, dense MLP, and expert layers, raised AIME 2025 accuracy from 42.9% to 64.2% while mean output length fell from 3,188 to 2,197 tokens. Only adapter weights are synchronized with SGLang in the LoRA recipe, and the optimizer state stays in GPU memory, removing the need for NVMe optimizer offloading.
What Hugging Face built with Beam
Hugging Face ran three demonstration projects to show what Beam can do as a coding and agentic model.
A native macOS dictation app
Beam was asked to build BeamSpeak, a native macOS menu bar dictation app with a compact recording pill and no main window or account. The app uses ⌘⇧Space to start and stop microphone recording, shows elapsed time, microphone level, and transcription status, and runs speech recognition entirely locally using Parakeet v3 through MLX, keeping the model loaded between recordings. It pastes the transcript into the original app if that app remains focused and Accessibility permission is granted, otherwise copies it to the clipboard without pressing Enter. The build used Beam with the Pi coding harness; the saved run includes follow-up prompts, and the final source also includes later Codex repairs.
Multi-agent scaling on ProgramBench
To test whether Beam makes faster progress by splitting a programming task across agents, Hugging Face paired Beam with the open-source Pi coding agent and compared three setups: one agent working alone, five persistent peers collaborating through a shared Git repository, and a lead agent delegating work to asynchronous subagents. The harness ran on Hugging Face jobs with inference through Reflection's platform API.
The tasks came from ProgramBench, where agents must recreate a program from its compiled executable and documentation without seeing its source code or the evaluator's tests. Each setup had the same 8M API-token ceiling and two-hour solver limit per task, shared across its agents. The published results retain all 15 original episodes, though the current figure shows a post-hoc subset of jq, xsv, FIGlet, and hexyl; ripgrep was excluded across all three setups because grading its saved artifacts was too slow. The time axis uses fixed reference rates for model tokens plus measured tool time, rather than elapsed wall time.
A fine-tuned incident-triage decision model
Hugging Face tested whether a coding agent could turn an incident-triage policy into a small, fast decision model. The task was to read a developer incident and choose the responsible team, its severity, and whether someone should be pinged immediately. The workflow started with Beam, then continued with GLM-5.3 in Pi after Beam's quota ran out.
Using Unsloth, the team fine-tuned Qwen3.5-0.8B to score these choices directly. The workflow first built a synthetic dataset of 2.3k incidents split by incident cluster for training, calibration, and testing, then trained a LoRA adapter and decision head for three epochs with a 4-bit backbone on one H100 80GB on the Science Slurm cluster. As a baseline, the unadapted model was prompted with the full triage policy and ticket, without examples, and asked to generate the three decisions as JSON. On the same 320 synthetic test tickets, the fine-tuned decision model got all three answers correct on 76.25%, versus 4.69% for prompted zero-shot, and mean ticket latency fell from 1,218.06 ms to 102.58 ms — an 11.9× speedup.
Build with Beam: the $10,000 Hub credits program
Reflection is giving out $10,000 in Hugging Face Hub credits to the best projects that use Beam. To enter, build a Space, model, or dataset repo and share it on the model's repo page. Suggested project types include a demo that uses Beam on Spaces, a quantization for your favorite hardware, a synthetic dataset for a specific problem, an RL environment built with or for Beam, or a benchmark of Beam with shared results.
What Beam means for open-model deployment
Beam's significance is that it puts frontier-scale parameters under an Apache 2.0 license with a deployment path that does not require a frontier-scale cluster. The combination of 23B active parameters per token, a KV cache that grows at roughly one sixth the naive rate, and an NVFP4 checkpoint at roughly 280 GB means a four-B200 configuration is the practical target for serving, with FP8 on H200s as the pre-Blackwell alternative.
The model is text-only and aimed at coding and agentic work rather than general multimodal use. Its tooling choices — a response template in the tokenizer, a dedicated tool-call grammar, five reasoning effort levels, and preserve_thinking for reasoning that persists across turns — are all oriented toward multi-step agent loops rather than single-turn chat.