HackerRank Hiring Agent: The Risks of Non-Deterministic AI Resume Screening

LLM-Based Resume Scoring is Inherently Non-Deterministic

Using Large Language Models (LLMs) to assign numerical scores to resumes results in high variance, meaning the same candidate can receive wildly different scores across multiple evaluations. In a test of the open-source hiring-agent tool by HackerRank, a single resume scored between 66 and 99 out of 100 across 100 runs using the gemma3:4b model at a low temperature of 0.1. This inconsistency means that if a company sets a hard cutoff score (e.g., 85), a qualified candidate could be rejected simply due to the stochastic nature of the model's output.

Analysis of the HackerRank Hiring Agent Architecture

The hiring-agent tool operates by parsing a PDF resume into text and using an LLM to extract structured data across six categories: basics, work history, education, skills, projects, and awards. It further enhances context by scanning the candidate's GitHub profile and top repositories before performing a final grading based on a weighted rubric:

• Open Source Contributions: 35 points
• Personal Projects: 30 points
• Work Experience: 25 points
• Technical Skills: 10 points
• Bonus Points: Up to 20 points (for startup experience, technical blogs, etc.)

Consistency vs. Utility in Scoring

Evaluation of the tool shows a stark divide between "checklist" tasks and "judgment" tasks:

• High Consistency, Low Utility: Technical skills scoring is highly consistent because it functions as a keyword checklist (e.g., "Does the resume mention React?"). Similarly, work experience often returns a perfect score regardless of the candidate's actual seniority because the prompt lacks a detailed rubric or anchors to differentiate a junior from a principal engineer.
• Low Consistency, High Noise: Project and open-source scoring exhibit massive variance. Because the LLM is asked to make subjective judgment calls on "architectural complexity" or "significance" without a rigorous framework, the resulting score is essentially a "vibe-check."

Critical Design Flaws in AI Screening

Technical analysis of the hiring-agent prompts and implementation reveals several systemic issues that compromise its reliability:

1. Over-Reliance on Proxies

The scoring rubric heavily weights open-source contributions and personal projects (65% of the base score). This biases the system against experienced engineers who may have spent decades building proprietary systems that cannot be shared on GitHub, effectively penalizing professional experience in favor of public-facing hobbies.

2. Prompt Underspecification

The evaluation prompts are often too brief to provide the LLM with a stable baseline. Without clear examples of what constitutes a "15-point" versus a "25-point" experience, the model defaults to arbitrary numbering. Furthermore, attempting to perform all evaluation steps in a single prompt rather than breaking them into modular sub-tasks increases the likelihood of hallucinations and inconsistent weighting.

3. The "Determinism" Fallacy

Setting the model temperature to 0 or 0.1 does not guarantee deterministic results. As noted in GitHub issue #35, non-determinism persists even at temperature 0. LLMs are stochastic processes; reducing temperature makes the probability distribution "spikier," but it does not transform a subjective judgment call into a repeatable mathematical calculation.

Industry Perspectives and Counterpoints

The release of this tool sparked significant debate among engineering leaders and candidates regarding the ethics and efficacy of AI screening.

The Volume Argument

Some hiring managers argue that despite the inaccuracy, AI screening is a necessary evil due to application volume. One commenter noted that even a 35% success rate in elevating candidates is preferable to an exhausted human reviewer missing qualified people in a pool of hundreds of applicants per hour.

Legal and Ethical Risks

Critics point out that such tools may be illegal in jurisdictions like the EU due to anti-discrimination laws. Because LLMs can apply systematic biases that are difficult to audit, using them for automated rejection creates significant legal liability for employers.

The Developer's Intent

The HackerRank CTO clarified that the hiring-agent was designed as a ranking tool for intern applications (50k-60k per year) rather than a rejection tool. He stated that the production version uses more capable Gemini models and that the system was intended to help humans decide which resumes to read first, not to replace human judgment.

Summary of Technical Recommendations

To move beyond "vibe-based" AI screening, technical editors and engineers suggest the following improvements:

• Comparative Judgment: Instead of absolute scoring, task the LLM with comparing two resumes and justifying a preference.
• Structured Output: Use forced-choice categorical options rather than open-ended numerical ranges to reduce variance.
• Modular Prompting: Separate the extraction of facts from the evaluation of those facts into distinct pipeline stages.
• Evidence-Based Scoring: Rather than relying on star counts, use tools to verify if claimed skills match actual code commits in a candidate's GitHub.