RK3588S YOLOv8n UAV Detection Pipeline

The RK3588S YOLOv8n pipeline achieves real-time UAV detection at 46 FPS, saturating the hardware sensor's ceiling. By offloading capture, resizing, and inference to fixed-function silicon, the system maintains a flat memory footprint of approximately 140 MB per stream, allowing it to run on budget 2 GB RK3588S boards.

Hardware-Accelerated Pipeline Architecture

To eliminate CPU bottlenecks and minimize RAM usage, the pipeline offloads every heavy per-frame operation to dedicated hardware blocks on the RK3588S SoC:

  • Capture: Handled by the Image Signal Processor (ISP).
  • Color Conversion and Resizing: Handled by the Rockchip RGA (Raster Graphic Acceleration) unit.
  • Inference: Executed on the Neural Processing Unit (NPU).

By using a fixed pool of pre-allocated buffers (BufPool) instead of per-frame allocations, the system avoids large intermediate framebuffers and scratch tensors on the CPU side. This results in a bounded Resident Set Size (RSS) of 137–152 MB for a single 1080p stream and 276–304 MB for two concurrent streams.

Maximizing NPU Throughput

Standard single-threaded loops typically achieve ~31.2 FPS with YOLOv8n at 640×640 resolution. This project increases throughput to 46 FPS by implementing a 3-thread inference pool. This is achieved by creating one RKNN context per NPU core using rknn_dup_context and rknn_set_core_mask, effectively pipelining the workload across all three available NPU cores.

Multi-Process Topology and LLM Integration

The system is designed as a chain of independent OS processes communicating via Unix-domain sockets. This modular approach allows for a composable pipeline where detections flow through several stages:

  1. Detection: The primary YOLOv8n pipeline.
  2. Tracking: A ByteTrack stage for multi-object tracking.
  3. Temporal Features: An extraction stage for analyzing movement over time.
  4. Presence FSM: A Finite State Machine to manage object presence.
  5. LLM Summary: An on-demand natural-language assessment using Qwen2.5-0.5B.

To integrate the Large Language Model (LLM) without crashing the vision pipeline, the system employs a blackout/resume control plane. This mechanism temporarily frees the NPU from camera tasks, allowing the LLM to run at full speed before handing control back to the vision pipeline.

Build and Deployment

The project supports both native builds on the board and cross-compilation from x86-64 Linux or WSL using a provided toolchain. The required SDK components include librga v1.10.5_[8] and librknnrt v2.3.2.

Deployment Commands

# Native build
cd yolov8n_cap_multithread
bash build.sh

# Execution
./yolov8n_cap_multithread <rknn model> <device number> <rtsp port | hdmi>

Community Insights and Technical Discussion

While the project demonstrates high efficiency on edge hardware, some technical discussions highlight different approaches to NPU optimization. One contributor suggested that increasing batch size is the standard method for increasing throughput, rather than multithreading job submissions. The author clarified that the primary goal was to explore the limits of the RK3588S vision pipeline by keeping as much processing as possible off the CPU.

"The main trick is not the YOLO model itself, but the pipeline structure: MIPI capture through the ISP, resize/color conversion through RGA, and YOLOv8n inference through all 3 NPU cores with one RKNN context per core."

The project is licensed under Apache License 2.0 and is intended for educational and research purposes.

Sources