Ripgrep 15.2.0 musl Binaries Segfault During Large Searches

Ripgrep musl Binaries Experience SIGSEGV Crashes

Ripgrep version 15.2.0 binaries built for the x86_64-unknown-linux-musl target occasionally crash with a SIGSEGV (segmentation fault) when performing very large searches with a high degree of concurrency. The crash occurs during a calloc call initiated by opendir, specifically triggering an integrity assertion failure regarding heap metadata within musl's mallocng allocator.

Technical Root Cause and Backtrace

The crash is triggered when ripgrep traverses very large directory trees—specifically those containing millions of files (e.g., 1.8 million files across 20GiB of data). The failure manifests as a crash in get_meta() within mallocng/meta.h, indicating that the heap metadata has become corrupted or inconsistent.

The execution path leading to the crash is as follows:

  1. ignore::walk::Worker::run initiates a directory traversal.
  2. std::fs::read_dir is called to list directory contents.
  3. This leads to a call to the C library function opendir.
  4. opendir calls calloc to allocate memory for the directory stream.
  5. mallocng (musl's allocator) detects an integrity violation in the heap metadata and triggers the segmentation fault.

Reproduction Requirements

Reproducing the bug requires a specific set of conditions involving scale and concurrency:

  • Target Binary: A binary built for x86_64-unknown-linux-musl (e.g., the official 15.2.0 release binary).
  • Scale: A directory tree with a massive number of files. A reproduction script (generate_repro_tree.py) was used to create a tree with approximately 1.8 million files and 20GiB of data.
  • Concurrency: High CPU core counts (e.g., 24 cores) and high concurrency levels.
  • Environment: The issue was observed on OpenSUSE Tumbleweed Linux x86_64.

Community Insights and Analysis

Discussion surrounding the issue highlights that the root cause extends beyond ripgrep itself, pointing toward a deeper interaction between the musl libc and the Linux kernel.

Kernel-Level Bug

Evidence suggests that this issue is actually a manifestation of a kernel bug. Community members referenced a kernel patch and a detailed analysis linking the crash to kernel-level memory management rather than a logic error within ripgrep's Rust code.

Allocator Performance and Contention

Some users noted that mallocng in musl is known to struggle with contention during high-concurrency multithreading. One contributor suggested that replacing the default musl allocator with a more performant alternative like mimalloc can significantly reduce contention and potentially avoid these types of failures in high-performance applications:

"mallocng is bad at dealing with contention during multithreading... Switching to mimalloc improved performance by 20x... it should have never surfaced this way in the first place."

Filesystem Impact

From an operational perspective, running high-concurrency searches across massive cluster filesystems can be detrimental to the filesystem's metadata mechanisms. One user warned that this pattern of small I/O generates extreme pressure on metadata servers, which can bring high-bandwidth filesystems to their knees.

Sources