Understanding FreeBSD Memory Reporting and the ZFS ARC Cache

Understanding FreeBSD Memory Reporting and the ZFS ARC Cache

Memory Reporting Discrepancies in FreeBSD

Memory reporting in FreeBSD often appears inconsistent across different system monitoring tools because each tool employs a different heuristic to define "used" versus "available" memory. This is primarily due to how the operating system manages its Virtual Memory (VM) system and how it utilizes RAM for disk caching to optimize performance.

In FreeBSD, memory is categorized into several page queues:

  • Active: Pages currently being used by userland processes.
  • Inactive: Pages not accessed for some time; these are reclaimable and can be freed if the system needs more memory.
  • Laundry: Pages queued to be written to swap.
  • Wired: Memory used by the kernel or locked processes that cannot be paged out. This category also includes the ZFS Adaptive Replacement Cache (ARC).
  • Free: Purely unused memory.

Because inactive pages and parts of wired memory (like the disk cache) can be reclaimed by the kernel, tools must decide whether to count these as "used" or "free."

The Role of ZFS ARC in Memory Usage

ZFS, the default filesystem for most modern FreeBSD installations, uses the Adaptive Replacement Cache (ARC) to store recently used data in RAM. Unlike standard kernel caches, ARC bypasses the traditional VM system and is stored within the "wired" memory category.

Because ARC is designed to shrink automatically when the system demands more memory for applications, it is effectively reclaimable. However, if a monitoring tool simply sums "active" and "wired" memory, the ARC cache will make the system appear to be using significantly more RAM than is actually required by running processes.

To accurately track ARC usage, administrators can use sysctl to query specific kernel parameters:

  • kstat.zfs.misc.arcstats.size: Current cache size.
  • kstat.zfs.misc.arcstats.c_min: Minimum configured cache size.
  • kstat.zfs.misc.arcstats.c_max: Maximum configured cache size.

Analysis of Tool Heuristics: btop, fastfetch, and htop

Different tools calculate memory usage using different formulas, leading to conflicting reports on the same machine.

btop

Historically, btop used a simplified heuristic: used memory = active + wired. This approach is often inaccurate on FreeBSD because it ignores the reclaimable nature of the ARC cache and fails to account for inactive pages.

fastfetch

fastfetch utilizes a more inclusive definition of free memory: free memory = free + inactive + cache. Consequently, used memory = total - free memory. This generally provides a more optimistic view of available resources.

htop

htop typically reports memory categories separately but calculates used memory as: used memory = wired + active + laundry.

Critical Bugs in btop Memory Reporting

Research into the btop source code (src/freebsd/btop_collect.cpp) revealed two significant issues affecting FreeBSD users:

32-bit Integer Overflow

btop stored memory page counts in u_int (unsigned 32-bit integers). When calculating total bytes by multiplying the page count by the page size, any value exceeding 4GiB (4,294,967,296 bytes) would wrap around. For example, a system with 4.42 GiB of used memory might be reported as having only 422 MiB used due to this integer overflow.

Legacy Cache Reporting

btop attempted to fetch cache data from vm.stats.vm.v_cache_count. However, this sysctl parameter has been a "dummy for compatibility" since FreeBSD 12.0 and always returns 0. The actual cache queue has not existed since FreeBSD 6.3.0.

Implementing Correct Memory Tracking

To fix these discrepancies, memory monitoring tools must explicitly account for the ZFS ARC and the FreeBSD buffer cache (vfs.bufspace).

Corrected logic for calculating used memory involves:

  1. Calculating Total Cache: Summing vfs.bufspace and the variable portion of the ARC cache (current size minus minimum size).
  2. Calculating Raw Used: Summing active and wired memory.
  3. Calculating Actual Used: Subtracting the calculated cache from the raw used memory: usedBytes = (cachedBytes < rawUsed) ? rawUsed - cachedBytes : 0.

These improvements have been submitted as pull requests to btop and fastfetch (where the logic was extended to all ZFS-supporting systems, including Linux and NetBSD) and htop to ensure consistent and accurate memory reporting across the FreeBSD ecosystem.

Sources