Qwen3-8B acceleration on Intel Core Ultra with depth‑pruned draft models

TL;DR

Accelerated Qwen3-8B on Intel® Core™ Ultra using speculative decoding and a depth‑pruned Qwen3‑0.6B draft achieves ~1.4× speedup, enabling fast local AI agents via 🤗 smolagents.

Qwen3‑8B: A Model Built for Agentic Workflows

Qwen3‑8B is a recent release in the Qwen family that includes native agentic behaviors such as tool invocation, multi‑step reasoning, and long‑context handling. These capabilities make it a natural fit for AI‑PC (AIPC) scenarios where an LLM must generate “thinking‑aloud” traces and intermediate steps, which increase token usage and demand high inference throughput. When paired with agent frameworks like 🤗 smolagents, QwenAgent, or AutoGen, Qwen3‑8B can power applications that call APIs, write code, and perform complex reasoning.

Accelerating Qwen3‑8B on Intel® Core™ Ultra with Speculative Decoding

Speculative decoding (see arXiv:2211.17192) speeds up autoregressive generation by letting a smaller draft model propose multiple tokens in a single forward pass, which the larger target model then validates. In the Intel Core Ultra benchmark:

  • Target model – Qwen3‑8B quantized to INT4 with OpenVINO (OpenVINO 2025.2).
  • Draft model – Qwen3‑0.6B quantized to INT8 with OpenVINO.
  • Hardware – Intel Lunar Lake integrated GPU (Arc 140V) on a Core Ultra 7 268V, 32 GB DDR5.

This configuration delivered an average 1.3× speedup over the baseline 4‑bit OpenVINO inference.

from openvino_genai import LLMPipeline, draft_model

target_path = "/path/to/target/Qwen3-8B-int4-ov"
draft_path  = "/path/to/draft/Qwen3-0.6B-int8-ov"
device = "GPU"

model = LLMPipeline(target_path, device,
                    draft_model=draft_model(draft_path, device))

streamer = lambda x: print(x, end="", flush=True)
model.generate(
    "What is speculative decoding and how does it improve inference speed?",
    max_new_tokens=100,
    streamer=streamer,
)

Note: Both target and draft models must be converted to OpenVINO format. Pre‑converted models are available from the links in the blog post, or you can follow the OpenVINO export guide.

Pushing Performance Further with Depth‑Pruned Draft Models

The theoretical speedup of speculative decoding is

$$ \text{Speedup}=\frac{E[#\text{generated tokens}]}{\gamma c + 1}, $$

where (\gamma) is the average number of tokens the target generates per step, (c) is the latency ratio between target and draft, and the numerator reflects the expected number of tokens produced per speculation window. Reducing draft latency (lower (c)) therefore improves overall speed.

Layer‑wise Pruning of the Draft Model

Recent work shows that model depth dominates inference latency. Inspired by layer‑wise compression research, the authors measured angular distance between layer outputs to identify low‑impact blocks and removed them. Six of the 28 layers in Qwen3‑0.6B were pruned, after which the draft was fine‑tuned on synthetic data generated by Qwen3‑8B from 500 k prompts in the BAAI/Infinity‑Instruct dataset.

Resulting Gains

The depth‑pruned draft model (available as OpenVINO/Qwen3-pruned-6L-from-0.6B-int8-ov) increased the speculative decoding speedup to ~1.4× versus the baseline, confirming the expectation that a faster draft yields higher overall throughput.

The full notebook reproducing these steps is linked in the post: https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/supplementary_materials/notebooks/qwen-3/qwen3.ipynb.

Integration with 🤗 smolagents

To demonstrate practical impact, the accelerated Qwen3‑8B + pruned draft pair was integrated into the 🤗 smolagents library. Developers can now instantiate a local AI agent that:

  1. Calls external tools (e.g., web search).
  2. Executes Python code (e.g., generating PowerPoint slides via python-pptx).
  3. Handles long‑context reasoning efficiently on Intel Core Ultra hardware.

The demo task asked the agent to summarize the key features of the Qwen3 series and produce a slide deck. The workflow combined a web‑search tool and a Python interpreter, illustrating how accelerated agentic models enable end‑to‑end AI applications on consumer‑grade hardware.

The same model pairing can be used with other agent frameworks such as AutoGen or QwenAgent, broadening the ecosystem for fast, local AI agents.

References

  1. Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., & Roberts, D. A. (2025). The unreasonable ineffectiveness of the deeper layers. Poster presented at ICLR 2025. https://arxiv.org/abs/2403.17887

Performance disclaimer – Benchmarks were performed with OpenVINO 2025.2 on an Intel Core Ultra 7 268V (2.20 GHz) with an integrated Arc 140V GPU and 32 GB DDR5. Results may vary with different configurations.

Sources