SJTUjinmao/ArkEval

What it solves

ArkEval provides a standardized, executable benchmark for automatically repairing code in ArkTS and OpenHarmony projects. It addresses the lack of evaluation signals for low-resource languages by transforming scattered real-world issues into a structured dataset that allows researchers to compare LLM-based code repair agents and provide feedback for training and reinforcement learning.

How it works

The project implements a full repair pipeline consisting of several stages:

1. Localization: It uses ArkTS-aware structural splitting and Qwen3-Embedding-8B with Milvus to retrieve candidate files from the codebase.
2. Filtering and Completion: A repair model selects core files and completes dependencies to define a constrained repair scope.
3. Patch Generation: The model generates a unified Diff patch, optionally utilizing a RAG system that provides official Huawei/OpenHarmony syntax and example code.
4. Evaluation: The system evaluates the patch using a strict loop: it checks for patch applicability, Hvigor compilation, and the execution of reproduction tests to determine Compile@1 (build validity) and Pass@1 (behavioral correctness).

Who it’s for

Researchers and developers working on LLM4Code, automated program repair (APR), and specifically those targeting the ArkTS/OpenHarmony ecosystem.

Highlights

• Real-world Dataset: Contains 502 executable repair instances from 9 public repositories and 149 OpenHarmony apps.
• Behavioral Validation: Distinguishes between a patch that merely compiles and one that actually fixes the bug via behavioral reproduction tests.
• Strict Test Construction: Tests are verified through a multi-agent audit and expert review to ensure they fail on the bug and pass on the fix.
• Comprehensive Pipeline: Includes tools for ArkTS-aware splitting, embedding-based retrieval, and automated execution evaluation.