Measuring code quality with AI is harder than it looks.

The pitch you have already seen

In the standard LLM-as-judge setup a foundation model reads a file or a diff and returns a paragraph, a score, or a pass/fail. The model decides what counts, and how much it counts, and the product trusts that judgement.

This works when the model is right. The literature suggests the model is often not right, and the marketing rarely says so. Four mechanisms keep this category quietly unreliable.

One. The judge believes its own reading of the code

A function called sanitize_inputthat does not sanitise reads as a sanitiser to a judge looking at code. The name does the work the code does not. The same surface-form sensitivity that lets a model be fooled by reassuring documentation gets it fooled by reassuring identifiers, function shapes, and helpful-looking comments. The standard fix is “prompt the model better”. This usually nudges the failure mode rather than removing it.

Facet’s answer is structural, not prompt-level. The judge cannot grade. It extracts atomic features with a one-clause basis and a line citation, and deterministic code computes the score from those features. A function name is a claim. A claim without cited evidence is absent. The opinion lives in version-controlled, deterministic logic, not in a model output.

Two. One judge for all the dimensions

Code quality is not one thing. Performance, security, readability, and audit trail are different questions that need different evidence. Picking one frontier model to grade all of them is a convenience, not a finding.

When we qualified judges per construct on the 14-dimension rubric, no single model came out on top everywhere. The judge with the highest overall agreement on one construct failed the minority-recall gate on another; a different open-weight judge caught the rare class there but ran weaker elsewhere. Different judges win different constructs in the production routing. Facet routes each dimension to the judge that earned the right to score it.

Three. Agreement without minority-class recall is the wrong number

The standard reliability number for LLM judges is agreement with a reference set. A judge that says “looks fine” to every healthy file is right most of the time because most files are healthy. The number looks great. The judge has caught nothing.

The criterion that bites is minority-class recall. Of the genuinely broken cases, how many did the judge catch? Facet’s reliability gate enforces it at ≥ 0.50 on every dimension. It has rejected high-agreement judges in favour of lower-agreement, higher-recall ones more than once. The label means something because the threshold is documented and the rejection is real.

Four. Temperature zero is not deterministic

The same code submitted twice can produce different scores. Provider routing, hidden quantisation differences between backends, and silent model updates all push this. Even at temperature 0 with the same seed, different backends of nominally the same model give different greedy outputs.

We measured the drift on real code and shipped four levers to tame it. A single pinned provider per call. A fixed seed. A chunked feature list when a long prompt diluted per-feature attention. An N-vote ensemble on the noisier dimensions. The label “reliable” on a Facet dimension requires a real-code feature-stability of at least 0.90 across re-runs. The instrument is allowed to drift. The claim attached to it is not.

What the research literature actually says

The marketing claims and the academic findings have diverged. A brief tour, in case you want to read source. The foundational work is well-known, but the 2025 and 2026 papers are where the field has moved to actually quantifying how unreliable the standard setups are.

The foundations from 2023 and 2024.

• Wang et al. (2023), Large Language Models are not Fair Evaluators. Formalises that an LLM judge’s preference between two candidates depends substantially on which one was presented first. The effect is large enough to flip verdicts, and prompt-level mitigations only partially help.
• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Documents position bias, verbosity bias, and self-enhancement bias across standard benchmarks. The de facto reference for the failure modes the category needs to address, not just acknowledge.
• Panickssery, Bowman & Feng (2024), LLM Evaluators Recognize and Favor Their Own Generations. Shows judges systematically prefer outputs from their own model family, with the preference correlating with how well the judge recognises its own writing. The implication for a one-judge-everywhere setup is direct.

What 2025 added.

• Shi et al. (2024, updated November 2025; AACL-IJCNLP 2025), Judging the Judges. A Systematic Study of Position Bias in LLM-as-a-Judge. 15 judges, 22 tasks, more than 150,000 evaluations. Position bias is systematic rather than random, the strength varies substantially by judge and by task, and solution-quality gap influences the bias more than prompt length. The most thorough position-bias work to date.
• Chen et al. (2025), Beyond the Surface. Measuring Self-Preference in LLM Judgments. Introduces a DBG score that compares judge-assigned scores to gold judgments to isolate self-preference from genuine quality differences. Shows the bias is real even after quality-adjustment and persists across model sizes, versions, and post-training data.
• Yamauchi, Yano & Oyamada (2025), An Empirical Study of LLM-as-a-Judge. How Design Choices Impact Evaluation Reliability. Evaluation criteria are critical for reliability, non-deterministic sampling improves alignment with human preferences over deterministic evaluation, and chain-of-thought reasoning offers minimal gains when clear evaluation criteria are already present. A counter-intuitive empirical result that argues for explicit rubrics over fancier prompting.
• Li et al. (2025), LLMs Cannot Reliably Judge (Yet?). A Comprehensive Assessment on the Robustness of LLM-as-a-Judge. 15 attack methods, 7 defences, 12 models. Robustness varies substantially by prompt design with differences reaching up to 40 percentage points. The title is the headline.
• Wang et al. (2025), MCTS-Judge. Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation. The code-specific paper. Reports that the LLM-as-judge paradigm in code correctness evaluation faces critical challenges including bias, misalignment, and fairness concerns, and improves accuracy from 41% to 80% by decomposing the judgement into multi-perspective sub-evaluations.

None of these papers say LLM-as-judge cannot work. They say it is harder than the marketing claims, the failure modes are specific and addressable, and the right response is measurement discipline rather than a louder model.

What Facet does differently. Measurement discipline, not a model upgrade

We treat the rubric as a psychometric instrument and ask it to pass instrument-grade tests. The pieces.

• Find then decide. The model never assigns a score. It extracts atomic features with a present, absent, or not-applicable status and a one-clause basis. Deterministic code does the scoring. The opinion lives in deterministic, version-controlled logic.
• Claims are not evidence. Reassuring names, comments, and docstrings are claims. Without cited line-level evidence, a claim is absent. This is the structural defence against the most common failure mode in the category.
• Four-bar reliability gate per dimension.Schema coverage ≥ 0.80, surface-weighted agreement ≥ 0.60, minority-class recall ≥ 0.50, and test-retest reproducibility ≥ 0.90. Documented, enforced, and the criterion that rejects high-agreement judges with poor minority-recall.
• Per-construct routing. Each of the 14 dimensions runs against the judge that earned the right to score it. Five different open-weight judges win five different constructs in the data.
• Formative scoring, fitted to evidence. A within-dimension unidimensionality test on a 47-file real-code corpus showed 13 of 14 dimensions are formative composites, meaning features that constitute the construct rather than reflect a single latent trait. A cumulative Guttman ladder over independent features mis-scores. A graded-response IRT model assumes the very unidimensionality the data deny. We use compensatory counting for the 13 and cumulative for the one dimension where the data supports it.

• Two sub-scores per dimension. Capability (positive indicators present) and violation (negative-polarity indicators present), reported separately. A feature-rich-but-vulnerable file and a minimal-but-clean file are different things and do not collapse to one number.
• Determinism levers.Provider-pinned calls, fixed seed, chunked extraction on long prompts, and an N-vote ensemble on the noisier judges. The label “reliable” requires real-code feature-stability of at least 0.90, not temperature-0 hope.

The full per-dimension evidence and the Lane A construct-validity results live on the methodology page. To watch the whole discipline applied end-to-end to a single dimension, external weights, adversarial self-review and all, read the Security worked example.

What we do not claim

• Construct validity is partial.Lane A (exploratory, single open-weight judge, N ≈ 47 to 50 real files) is complete. No inter-dimension pair |r| ≥ 0.70. Roughly 6 weakly-correlated factors with no general-quality halo. 4 of 4 known-groups discrimination on the dimensions whose absolute level was least certain. Lane B confirmatory work with a second independent judge, a larger held-out corpus, formal inter-rater κ, and formal DIF across languages is in progress.
• One judge per dimension is still a simplification. Ensemble extraction on the noisier dimensions partly mitigates run noise. It does not give us cross-judge convergent validity. That is on the Lane B list.
• Functional correctness is scoped out. Facet measures how code is written, not whether it computes the right answer. Broken code with the right machinery can still score well on the structural dimensions. We say this plainly rather than pretend the rubric covers it.
• The predicates are not published. The exact indicator predicates and per-feature criteria do not appear on the methodology page or in this article. The rubric is the product. Third parties can audit it under collaboration agreement.

The honesty frame

AI-powered insights without measurement discipline is a confident model. With measurement discipline, you get an instrument a skeptic can test. The gap between the two is not a better foundation model. It is the work the category has mostly not done.

The full evidence and per-dimension data live on the methodology page. The thesis in action is on the demo page, where the same small problem is solved several ways and profiled by the live engine. None is best on every axis. That is what a profile, not a score, means.

Profile your codeRead the methodology