Bringing Robotics AI to Embedded Platforms: Dataset Recording, VLA Fine-Tuning, and On-Device Optimizations

Hugging Face and NXP have detailed a systems engineering approach to deploying Vision-Language-Action (VLA) models on embedded robotic platforms. The primary takeaway is that successful deployment requires a combination of high-consistency dataset recording, architectural decomposition of the model graph, and asynchronous inference to overcome the compute and memory constraints of edge hardware.

High-Consistency Dataset Recording

Reliable VLA policy learning depends more on data consistency than data volume. For a task such as placing a tea bag in a mug, the following recording practices are critical for maintaining accuracy:

Environmental and Hardware Stability

  • Fixed Camera Mounts: Rigid mounts are required to prevent pose drift; camera shifts caused by robot vibrations or environment resets lead to severe accuracy loss.
  • Controlled Lighting and Contrast: Fixed light sources and the avoidance of sunlight prevent temporal variance. High contrast between the robot arm, the object, and the environment is necessary to avoid "white on white" scenarios.
  • Calibration Backups: Maintaining backups of robot and teleoperator calibrations prevents the need to re-record episodes after software crashes.
  • Input Restriction: Operators must rely exclusively on the camera inputs available to the policy at runtime to avoid introducing information the model cannot access during inference.

Viewpoint Strategy

Using a mix of viewpoints improves global accuracy. NXP recommends a three-camera setup:

  • Top View: Provides a global view of the scene.
  • Gripper Camera: Highly recommended for fine manipulation tasks as it provides the closest view for precise grasps and alignment.
  • Left View: Complements the top view for height and depth perception.

Physical and Data Diversity

  • Hardware Tweaks: Applying heat-shrink tubing to gripper claws increases friction and reduces slippage, improving policy learning stability.
  • Workspace Clustering: The reachable workspace should be divided into starting-position clusters (e.g., 10x10 cm), with at least 10 episodes recorded per cluster.
  • Recovery Episodes: Including "recovery episodes" (roughly 20% of the training set) where the robot must retrieve an object after a failed first attempt improves the overall success rate.
  • Validation Splits: A dedicated cluster (e.g., cluster 6) should be removed from the training set to serve as an unseen validation set to prevent overfitting.

Fine-Tuning VLA Policies

Practical fine-tuning for the "tea bag in mug" task involved 120 episodes across 10 clusters, using three 640x480px cameras at 30fps.

Key findings regarding model checkpoints include:

  • ACT (Action Chunking with Transformers): The best trade-off between accuracy, generalization, and smoothness was found between 100k and 160k training steps (with 100 actions per chunk).
  • SmolVLA: This model requires significantly more training steps (with 50 actions per chunk) to reach a similar trade-off.
  • Evaluation Metric: Final checkpoints should be selected based on success rates on both training and validation sets rather than relying solely on training loss.

Optimizing for the NXP i.MX 95 SoC

The NXP i.MX 95 SoC integrates Arm Cortex-A55 and Cortex-M cores, a Mali GPU, and the eIQ Neutron NPU. To enable efficient edge inference, NXP employs three primary strategies:

Architectural Decomposition

Instead of a monolithic graph, the VLA is partitioned into logical sub-blocks:

  1. Vision: Processes RGB frames into visual embeddings.
  2. LLM Backbone: Generates action tokens from visual and textual embeddings.
  3. Action Expert: Applies flow matching to denoise action samples into final control commands.

This separation allows for per-block quantization and independent scheduling, specifically allowing the action expert to run at a lower frequency.

Strategic Quantization

Quantization impact varies by block:

  • Vision Encoder and LLM Prefill: These blocks show limited accuracy degradation when quantized.
  • Action Expert: Quantization of the denoising flow significantly degrades performance because errors accumulate across iterative denoising steps. Consequently, this block is kept at higher precision to maintain stability.

Asynchronous Inference and Scheduling

Synchronous control loops create idle gaps and oscillatory corrections because the robot remains idle while the model runs inference. Asynchronous inference dissociates generation from execution, allowing the robot to execute the current action chunk while the next chunk is computed in parallel.

For this to be effective, the end-to-end inference latency must be shorter than the action execution duration (inference time < execution time).

Performance Benchmarks on i.MX 95

Testing on the "tea bag in mug" task with a test set of 20 episodes and a validation set of 10 episodes yielded the following results:

Policy Format Inference Latency Global Accuracy (30 episodes)
ACT ONNX FP32 2.86 s 0.96
ACT Optimized 0.32 s 0.89
SmolVLA ONNX FP32 29.1 s 0.47

NXP has established an optimized on-board inference latency baseline for SmolVLA at 6.15 seconds, with future work focusing on further NPU optimization, simulation environments for scalable data generation, and Sim-to-Real transfer.

Sources