SIMD-Accelerated Computer Vision on ESP32-S3: Achieving High Throughput on a $2 Microcontroller
SIMD-Accelerated Computer Vision on the ESP32-S3
A SIMD-accelerated FAST feature detector was implemented for the ESP32-S3 microcontroller, leveraging its 128-bit SIMD instructions to achieve a throughput increase of approximately 220%. This optimization allows the low-cost microcontroller to process a 30 frames-per-second (fps) VGA stream, demonstrating the viability of real-time computer vision on affordable edge hardware.
Performance and Hardware Context
The ESP32-S3 is a powerful microcontroller for its price point, featuring a dual-core CPU running up to 240MHz, along with integrated WiFi and Bluetooth Low Energy radios. The implementation of the FAST feature detector utilized the chip's limited but available SIMD instructions to significantly boost processing speed.
In testing, the SIMD-accelerated implementation processed a QVGA (320x240) frame in approximately 6 milliseconds. The overall throughput improved from 5.1 Megapixels/second (MP/s) to 11.2 MP/s, representing a substantial performance gain for an embedded system.
Technical Challenges and Solutions
Developing the SIMD-accelerated FAST detector required working directly with the ESP32-S3's assembly language and navigating several specific hardware limitations of the Xtensa LX ISA.
Handling Misaligned Memory Access
The SIMD architecture does not support direct misaligned reads. When sampling pixel data in cardinal directions (east and west) around a central pixel, the required memory blocks were misaligned. This was solved by utilizing a specific chip instruction that allows for extracting a slice of two concatenated registers into a single register, enabling the extraction of the necessary pixel data.
Overcoming Signed vs. Unsigned Arithmetic
The ESP32-S3's SIMD instructions only implement comparisons for signed eight-bit numbers, while pixel data is stored as unsigned eight-bit numbers. Directly reinterpreting unsigned data as signed data causes non-linear mapping issues, particularly in the range of 128 to 255, which breaks standard comparison operations.
To resolve this, the pixel values needed to be shifted from the unsigned range [0, 255] to the signed range [-128, 127]. Standard subtraction was not possible due to the SIMD arithmetic operations being saturating, meaning values above 127 would clamp at -128 instead of underflowing.
This was overcome by exploiting the properties of two's complement arithmetic. Since $x - 128$ is equivalent to adding $-128$, and $-128$ is represented by the highest bit (0b10000000), the operation $x - 128$ can be achieved by XORing the unsigned byte $x$ with the mask $0 ext{x}80$ ($x ext{ XOR } 0 ext{x}80$). This technique allows the processor to treat the unsigned data as if it were in the correct linear range for comparison and arithmetic.
Implementation Details
The implementation involved writing custom assembly code, including a basic register allocator named basm, to manage the low-level operations. The FAST corner pre-test was optimized by designing a function to calculate the extrema count for sixteen pixels simultaneously, matching the capacity of the vector register.
Community Insights and Further Considerations
Technical discussions surrounding this project highlighted several points regarding the ESP32-S3's SIMD capabilities and alternative hardware:
- SIMD Difficulty: Several contributors noted that the SIMD instructions on the ESP32-S3 are complex and difficult to use effectively.
- Performance Metrics: One comment clarified that the reported increase from 5.1 MP/s to 11.2 MP/s is a 120% increase, not a 220% increase, as percentage increase is calculated relative to the starting value.
- Alternative Architectures: For users seeking more powerful SIMD, the CH32V407/CH32V467 was mentioned as an alternative, supporting the embedded subset of the RISC-V vector extension, which offers a wider SIMD ISA (64-bit width) compared to the ESP32-S3's implementation.