Chiaro SOC 2 Methodology Open Source Release

Chiaro has open-sourced its full SOC 2 readiness and audit methodology to replace the industry's reliance on "faith" with a transparent, machine-readable framework. By publishing the exact controls, evidence standards, and judgment calls used during examinations, Chiaro allows companies to run their own readiness assessments against the same bar used in actual audits.

Machine-Readable Audit Framework

The Chiaro methodology is delivered as a set of structured files designed for both human review and AI integration. This approach ensures that the testing layer is no longer a black box, allowing users to verify exactly how a control is tested and what constitutes a passing grade.

The repository includes the following core components:

framework/controls.json: A library of 86 specific controls.
framework/test_attributes.json: 355 test attributes defining what is tested, typical evidence, pass criteria for point-in-time and over-a-period assessments, and applicability rules.
framework/criteria.json: Mapping of controls to the 61 Trust Services Criteria.
framework/evidence_map.json: 22 evidence sources mapped to the controls they satisfy.
method/scoping-playbook.json: Guidelines for classifying systems as in-scope, out-of-scope, or as a subservice organization.
method/collection-rules.md: The operating rules for evidence collection.
method/tools.md: A list of every tool a connected AI can call, generated from the live server.

Calibrating AI Judgment with Worked Examples

To prevent AI models from grading themselves too leniently, Chiaro provides 498 calibration examples. These examples record a specific judgment call, the initial verdict reached by an AI, the correct verdict, and the reasoning behind the correction.

These examples are synthetic to protect client confidentiality but are based on real-world scenarios encountered during fieldwork. The distribution of these corrections reveals a key insight into AI auditing: 303 examples correct an AI that was too strict, while 195 correct one that was too lenient. According to the author, AI engines often over-flag issues because they cannot perceive when a missing artifact is covered by other evidence elsewhere in the system.

Shift from Sampling to Complete Populations

Chiaro's Type II testing method defaults to verifying complete populations rather than relying on statistical sampling. Because modern company data is machine-generated, it can be verified at machine speed.

Key aspects of this approach include:

Full Population Testing: The client's AI retrieves every single instance of a control's execution (e.g., every change, termination, or access review) within the observation window.
Corroboration: Completeness is verified through recorded retrieval and reconciliation against independent second sources where available.
Deterministic Testing: Structured evidence is tested deterministically, while prose is tested via calibrated reading, with all candidate deviations confirmed by a CPA.
Sampling as Fallback: Sampling is only used when a population cannot be retrieved in full. In these cases, selection is seeded from a hash of the banked population to prevent any party from steering or "re-rolling" the sample.

Implementation and Legal Constraints

While the methodology is open-sourced under the CC BY 4.0 license, there is a critical distinction between the methodology and the legal attestation. Anyone can use the Chiaro framework to prepare for an audit or run readiness checks, but only a licensed CPA firm can sign a SOC 2 opinion, as the signature is the regulated portion of the process under AT-C 205.

Industry Context and Transparency

The decision to open-source this methodology stems from a critique of the current SOC 2 landscape, where different firms may apply vastly different levels of rigor while producing identical-looking reports. By publishing the methodology before performing Type II examinations, Chiaro aims to ensure the rules are established before the results are known, preventing the "fitting" of rules to results after the fact.