During the autoregressive decode phase of Large Language Model (LLM) serving, generation throughput is fundamentally constrained by the memory bandwidth wall. For every single token generated, the inference engine must stream the entire parameter footprint of the model from High-Bandwidth Memory (HBM) or GDDR6 into on-chip registers and SRAM. On an Intel Arc Pro B70 GPU (608 GB/s GDDR6) running a 70-billion parameter model in 4-bit precision (~38.5 GB footprint), this memory traffic caps single-stream decode speed at approximately 15.8 tokens per second.

DFlash (Chen et al., ICML 2026 / arXiv:2602.06036) and its successor DFlash2 represent a fundamental architectural departure from traditional speculative decoding. Instead of relying on a sequential, token-by-token autoregressive draft model (such as EAGLE-3 or Medusa), DFlash deploys a non-autoregressive block diffusion drafter. By conditioning on deep context features extracted from the frozen target LLM and generating an entire block of candidate tokens in a single parallel step, DFlash eliminates sequential drafting latency, overcomes error accumulation, and multiplies decode throughput by 2.5× to 3.8× with 100% mathematical losslessness.

1. Executive Summary & Core Mechanics

Speculative decoding systems partition inference into two alternating phases:

1. Draft Phase: A lightweight model proposes a candidate sequence of γ tokens.
2. Verification Phase: The large target model evaluates all γ tokens concurrently in a single forward pass, accepting valid tokens and rejecting divergences via modified rejection sampling.

Classical speculative drafters (EAGLE-1/2/3, Lookahead, Medusa) generate draft candidates sequentially: predicting token t+k requires computing token t+k-1. Consequently, drafting latency scales linearly with speculation length:

To prevent drafting latency from overwhelming verification gains, classical drafters are restricted to extremely shallow architectures (often a single transformer layer). This constrained capacity limits draft quality and causes acceptance rates to saturate rapidly.

DFlash replaces the serial draft loop with a parallel block diffusion formulation:

Because parallel tensor execution on modern GPUs utilizes compute units with high efficiency, t_{parallel} remains largely invariant to block size for moderate lengths (B ∈ [4, 16]). This decoupling allows DFlash to employ deeper, more expressive draft architectures (5 to 8 layers) that yield higher acceptance lengths without latency penalties.

2. Hardware Physics & The Memory Bandwidth Wall

To establish why block diffusion speculative decoding delivers super-linear serving speedups, we analyze the operational intensity and memory access dynamics of transformer inference.

Operational Intensity Derivation

The arithmetic operational intensity I is defined as the ratio of floating-point operations executed to memory bytes transferred across the GPU bus:

During the Prefill Phase (processing an input prompt of length N), matrix-matrix multiplications (GEMM) dominate. For a model with P active parameters and prompt length N, computation scales as 2 · P · N FLOPs, while weight memory reads scale as P · b bytes (where b is the byte-width per parameter). The arithmetic intensity is:

For N = 2048 tokens and 16-bit precision (b=2), I_{prefill} ≈ 2048 FLOPs/byte, placing execution firmly in the compute-bound regime of the hardware roofline.

During the Decode Phase (generating a single token at batch size 1), matrix-vector multiplications (GEMV) dominate. Computation for one token is 2 · P FLOPs, while all P parameters must be transferred from global memory:

For 4-bit quantized weights (b = 0.5 bytes per parameter), the decode operational intensity is:

On an Intel Arc Pro B70 GPU with 130 TFLOP/s FP16/FP8 matrix compute and B_{mem} = 608 GB/s GDDR6 bandwidth, the operational intensity required to saturate the compute units is:

Because I_{decode} = 4.0 ≪ 213.8, the execution engine sits idle for >98% of clock cycles, waiting for weight bytes to arrive from GDDR6.

The Single-Stream Decode Speed Ceiling

Consider Llama 3.3 70B quantized to 4 bits via Intel AutoRound (W4A16), occupying M_{weights} = 38.5 GB of VRAM. The theoretical maximum single-stream decode speed T_{baseline} is strictly bound by memory bandwidth:

Speculative Decoding Speedup Formulation

In speculative decoding with speculation window γ, the expected number of accepted tokens per verification cycle is denoted by τ ∈ [1, γ + 1] (including the target model’s bonus token). The average latency per generated token L_{per-token} and speedup ratio η are:

where L_{target} is the per-token latency of standard autoregressive generation (L_{target} ≈ T_{verify}).

In DFlash, because candidate drafting occurs in a single parallel pass (T_{draft} = t_{parallel} ≈ 3.2 ms) and verification of γ = 5 tokens takes T_{verify} ≈ 64.1 ms, an average acceptance length of τ = 4.15 yields:

This represents a 3.90× theoretical speedup over the 15.79 tok/s hardware ceiling without altering the underlying silicon or quantizing weights further.

3. Limitations of Classical Speculative Decoding

Speculative decoding architectures have evolved through three prior generations, each exhibiting specific engineering constraints:

1. Independent Small Drafters (Leviathan et al., 2023)

Using an independent smaller model (e.g., Llama-3.2-1B drafting for Llama-3.3-70B) suffers from representation mismatch. The draft model maintains separate vocabulary tokenization and hidden spaces, leading to low acceptance rates (τ ≈ 1.8 - 2.2). Furthermore, running a 1B–3B model sequentially incurs substantial memory read overhead of its own.

2. Medusa (Cai et al., 2024)

Medusa attaches multiple linear heads to the final hidden state of the target model to predict tokens at offsets t+1, t+2, dots, t+k. However, because the heads predict independently without cross-token communication:

Joint probability decays exponentially with sequence length, causing draft accuracy to collapse beyond position k > 2 on structured programming and mathematical reasoning tasks.

3. EAGLE & EAGLE-2/3 (Li et al., arXiv:2401.15077, arXiv:2406.16858, arXiv:2503.01840)

EAGLE addresses token uncertainty by shifting speculation from token space to top-layer feature space. EAGLE-2 introduces confidence-calibrated dynamic draft trees. EAGLE-3 introduces training-time test algorithms to align feature representations.

Despite these improvements, EAGLE-3 remains fundamentally autoregressive. Drafting γ tokens requires γ sequential forward passes through the draft head. To maintain low drafting overhead, EAGLE drafters are constrained to a single transformer layer. When scaling γ ≥ 8, cumulative sequential latency (8 · t_{step}) offsets verification savings, capping practical speedups below 3×.

4. Monolithic Diffusion Drafters (DiffuSpec, SpecDiff-2)

Recent attempts to apply diffusion models to drafting (Li et al., 2025a; Sandler et al., 2025) utilized large pre-trained 7B parameter diffusion models. The memory traffic required to stream a 7B drafter during every decode step introduces severe latency overhead, offsetting speculative efficiency gains.

4. DFlash Architecture (ICML 2026)

DFlash resolves the drafting bottleneck by framing candidate generation as a conditional block diffusion process operating over continuous feature spaces, tightly conditioned on target model hidden representations.

4.1 Target Context Feature Conditioning via KV Injection

Large autoregressive target models implicitly encode long-range syntactic patterns and future token semantics within their intermediate representations. DFlash captures this information by extracting hidden states from 5 uniformly distributed layers:

These extracted feature vectors are concatenated across the channel dimension and projected into the draft model’s hidden dimension D:

where W_c ∈ R^{D × 5D}.

Rather than injecting H_t only at the input layer (which causes information dilution across deeper layers), DFlash injects H_t directly into the Key and Value projection caches of every draft transformer layer:

where H_d represents the draft tokens’ intermediate states. The target context features H_t act as invariant memory anchors. They bypass the draft model’s Query projections, output projections, and Feed-Forward Networks (FFNs), introducing minimal compute and parameter overhead.

Draft Model Layer Memory Overhead:
- Target Hidden Dimension D = 2048, 5 Extracted Layers
- Projection Matrix W_c in BF16: 2048 * (5 * 2048) * 2 bytes = 41.94 MB
- Activation Memory during Block Size 16 Speculation: < 400 KB

4.2 Shared Embeddings & LM Head

To prevent parameter explosion and preserve semantic alignment with the target model’s vocabulary space, DFlash freezes and reuses:

1. Target Embedding Layer (W_{emb}): Converts discrete anchor tokens to input representations.
2. Target Unembedding Head (W_{lm_head}): Maps output states to vocabulary logits.

Only the 5 to 8 intermediate draft transformer blocks and the lightweight projection matrix W_c contain trainable parameters.

4.3 Training Formulation: Random Masked Blocks & Loss Weighting

During training, DFlash aligns the diffusion drafter to target model trajectory via three specialized mechanisms:

1. Anchor-Conditioned Masked Block Construction

Rather than masking arbitrary spans, DFlash randomly samples clean anchor positions from target response sequences. For each anchor a, the subsequent B-1 tokens are replaced with learnable mask tokens [M]:

This formulation mirrors inference, where the draft model is initialized from the bonus token produced by the target model’s previous verification step.

2. Block-Diagonal Flex Attention Mask

To train multiple blocks within a single forward pass without causal contamination, DFlash uses a sparse block-diagonal attention mask. Tokens within block k attend bidirectionally to each other and to the target context feature H_t, but cannot attend to tokens in block j ne k.

3. Asymmetric Position-Dependent Loss Weighting

In speculative decoding, an error at position k invalidates all subsequent predictions k+1 dots B. To accelerate optimization on high-leverage early positions, DFlash weights the cross-entropy loss with an exponential decay factor:

where γ is a scale hyperparameter tuned to the block length (γ = 7 for B=16, γ = 5 for B=10).

4.4 Mathematical Losslessness Guarantee

DFlash preserves the exact output probability distribution of the target model M_{target}. During verification, the target model computes logits for all candidate positions in parallel. For each candidate token x_k proposed by the drafter:

If candidate x_k is rejected (r > α_k where r ~ U(0, 1)), the generation sequence is truncated at k, and a replacement token is sampled from the normalized residual distribution:

Under greedy decoding (T=0), token x_k is accepted if and only if:

There is zero quality degradation, zero perplexity increase, and zero drift in benchmark task accuracy.

5. DFlash2 Innovations: Local Depthwise Convolutions & Path Selection

While DFlash established the efficacy of parallel block diffusion, single-pass non-autoregressive drafting exhibited a known limitation: coherence decay over long blocks (B ≥ 6). When drafting extended token sequences in a single pass without step-by-step causal attention, inter-token dependencies between non-adjacent positions weaken.

DFlash2 introduces two architectural mechanisms that eliminate coherence decay:

1. Local Depthwise 1D Convolutions

DFlash2 embeds 1D depthwise separable convolutions with kernel width k ∈ {3, 5} between the draft attention and feed-forward sublayers. For a block of hidden states H ∈ R^{B × D}:

Because depthwise convolutions operate across the sequence dimension with linear complexity O(k · B · D), they enforce local n-gram token-to-token transition constraints without incurring the quadratic O(B^2) compute overhead of causal self-attention.

2. Dynamic Candidate Path Selector

Instead of emitting a single deterministic token block [t_1, t_2, dots, t_B], the DFlash2 diffusion head emits top-m candidate predictions for each position within the block.

A lightweight path scoring kernel evaluates transition probability density along graph trajectories:

The path selector prunes low-probability branches and submits the optimal candidate path (or dense verification tree) to the target model.

6. Empirical Benchmarks & Epistemic Grounding

To maintain strict scientific and engineering integrity, benchmark figures below explicitly separate physically measured hardware telemetry recorded on our testbed from theoretical scaling projections derived from published paper acceptance lengths (τ).

Task-Specific Acceptance Length Breakdown (τ)

The average accepted tokens per step (τ) across distinct task domains illustrates where block diffusion excels (grounded in DFlash ICML 2026 evaluation suites):

7. Hands-on Production Serving Recipes

7.1 Method A: Deploying DFlash2 in SGLang (sgl-kernel-xpu)

SGLang provides native support for DFlash through its RadixAttention memory manager and speculative overlap scheduler (Spec-v2).

Step 1: Environment & Dependency Installation

# Verify Intel Level Zero runtime and Xe GPU detection
xpu-smi discovery

# Install SGLang with Intel XPU acceleration kernel
pip install torch torchvision --index-url https://download.pytorch.org/whl/xpu
pip install sgl-kernel-xpu sglang

Step 2: Launch SGLang Server with DFlash2 Speculative Engine

python3 -m sglang.launch_server \
  --model-path /models/Qwen2.5-32B-Instruct \
  --speculative-draft z-lab/Qwen2.5-32B-DFlash2-Draft \
  --speculative-algorithm dflash2 \
  --speculative-num-steps 6 \
  --device xpu \
  --mem-fraction-static 0.88 \
  --tp 1 \
  --port 30000 \
  --host 0.0.0.0

Step 3: Client Verification Script (Python)

import openai

# Connect to local SGLang DFlash2 server
client = openai.Client(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen2.5-32B-Instruct",
    messages=[
        {"role": "system", "content": "You are a high-performance systems programming expert."},
        {"role": "user", "content": "Write a C++20 SYCL kernel demonstrating 2D matrix tiling into shared local memory."}
    ],
    temperature=0.2,
    max_tokens=1024,
)

print(response.choices[0].message.content)

7.2 Method B: Deploying DFlash2 in vLLM (vllm-project/speculators)

vLLM integrates DFlash block diffusion drafters through its vllm-project/speculators package and PagedAttention v2 allocation manager.

Step 1: Python Engine Initialization

from vllm import LLM, SamplingParams

# Configure Llama 3.3 70B AutoRound W4A16 with DFlash2 Block Diffusion Drafter
llm = LLM(
    model="/models/Llama-3.3-70B-Instruct-AutoRound-W4A16",
    quantization="auto_round",
    speculative_model="z-lab/Llama-3.3-70B-DFlash2-Draft",
    num_speculative_tokens=6,
    speculative_draft_tensor_parallel_size=1,
    use_v2_block_manager=True,
    device="xpu",
    gpu_memory_utilization=0.92,
    max_model_len=8192,
    trust_remote_code=True,
)

# Sampling parameters (DFlash guarantees mathematical equivalence under all temperatures)
sampling_params = SamplingParams(
    temperature=0.0,  # Greedy verification mode
    max_tokens=512,
    top_p=0.95
)

prompts = [
    "Explain how block diffusion avoids autoregressive error accumulation during speculative decoding."
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Generated Output:\n{generated_text}")

8. Actionable Expert Pro-Tips

9. Technical FAQ

Q1: Is DFlash2 strictly mathematically lossless compared to vanilla autoregressive decoding?

Yes. DFlash2 employs the standard speculative rejection sampling algorithm formulated by Leviathan et al. (2023). Every candidate token block produced by the diffusion drafter is evaluated against the exact target model probability distribution P_{target}(x mid x_{<k}). If a candidate diverges, it is rejected and resampled from the adjusted residual distribution P’(x). The output distribution is provably identical:

Perplexity, pass@1 coding accuracy, and reasoning benchmark scores remain identical down to numerical floating-point precision.

Q2: What is the exact VRAM footprint of the DFlash2 draft model?

The DFlash2 draft model consists of 5 to 8 shallow transformer layers and the target context projection matrix W_c. It reuses the target model’s frozen embedding table and language modeling head. The entire draft checkpoint occupies between 450 MB and 850 MB of VRAM. On a 32GB Intel Arc Pro B70 card hosting a 38.5 GB 70B model (quantized to 4-bit across a 4-card 128GB cluster or 32B model on a single card), the draft model adds less than 3% to total memory consumption.

Q3: How does DFlash2 handle non-greedy temperature sampling (T > 0)?

Under temperature sampling (T > 0, top-p, or top-k), DFlash applies temperature scaling to both the draft logits and the target verification logits before calculating the acceptance ratio:

On non-greedy settings (T=1.0), DFlash maintains high acceleration (e.g., 4.03× speedup on Qwen3-8B in Table 1 of the ICML 2026 paper), outperforming EAGLE-3 across all benchmark datasets.

Q4: How does DFlash2 integrate with multi-GPU Tensor Parallelism (TP=2, TP=4)?

In multi-GPU environments (e.g., 4× Intel Arc Pro B70 128GB cluster with TP=4), the large target model is sharded across all GPUs via standard row-parallel and column-parallel matrix partitions connected via Intel oneCCL AllReduce over PCIe 5.0.

The lightweight DFlash2 drafter can be executed in two modes:

1. Replicated Drafter (Recommended): The small ~600MB draft model is fully replicated on each GPU rank. Drafting executes locally without inter-GPU communication.
2. Synchronized Parallel Verification: During verification, all 4 GPUs execute the target model forward pass over the full candidate block [t_1 dots t_B] in parallel, synchronizing activations via oneCCL in a single collective step. This maintains full 3.45× speedup without PCIe interconnect bottlenecks.