Worked example

Security (D6): one dimension, investigated in the open

Facet measures code across fourteen quality dimensions. The indicator set behind each one is proprietary. We make a single deliberate exception and narrate the whole investigation of Security, chosen because its external grounding is the cleanest, so a sceptical engineer can see exactly what it takes to make one dimension defensible, then judge the other thirteen by the same standard even though their contents stay private. This page describes where Security stands right now, not a finished product; it will keep changing as the remaining validation work lands, and a draft manuscript of the full validation programme is planned for this site. Treat this page as its accessible preview.

We chose Security to open up because its content sits closest to public consensus already (OWASP, CWE, NVD CVSS, and static analysers like SonarQube, Bandit, Semgrep and CodeQL), so narrating it costs the least while the rigour on display is identical to every other dimension. Treat this as the floor of the effort behind one dimension, and extrapolate it across all fourteen. What we hold back, deliberately, is the exact indicator list, the per-indicator weights, the gate threshold and the score formula. Those are the reproducible asset. The method and the evidence below are not, and they are what earns trust.

1. What is measured, and who does the scoring

Security here means resisting adversarial and malformed input, protecting secrets, and defaulting to safe behaviour. A reliability-qualified language model reads the code and does one narrow job: it extracts observable indicators with a line citation for every call. It never assigns a score. Deterministic code decides the level from those findings. Two poles are kept apart and never netted: a capability pole (a protective practice is present) and a violation pole (a harmful pattern is present). A missing protection and a present defect are different facts, and are reported as different facts.

2. How the judge earned the job

The model that extracts security indicators was not picked off a leaderboard. For every dimension, judge selection is a competition: a pool of candidate open-weight models each sits the same exam, a held-out battery of minimal pairs built per indicator - a clean version and a subtly degraded version of the same code, where the only difference is the thing the indicator is supposed to see. A candidate must clear four bars at once on that battery: agreement with the answer key, minority-class recall (does it catch the rare broken case, not just wave healthy code through), schema compliance, and test-retest stability of at least 0.90. The granularity matters: selection is per dimension - each of the fourteen is routed to the model that proved most reliable on that construct, because no single model wins everywhere - while the qualifying evidence is per indicator, so a weak indicator can never hide inside a strong dimension. In the most recent expansion, 53 of 54 new indicators cleared that gate; the one that failed was removed, not kept for completeness.

Selection is only half the work. The other half is teaching the chosen judge to see: the extraction contract treats names, comments and docstrings as claims, not evidence (a function called sanitize_input that does not sanitise is absent); an indicator defaults to absent unless the code proves it; long feature lists are split into chunked extraction calls where a single long prompt measurably diluted per-indicator attention (a fix we validated before shipping, not assumed); and every call is provider-pinned and seeded, with a majority vote across repeated extractions mopping up residual flicker. When an indicator still cannot be detected reliably after one careful reword, it is dropped. We do not score what we cannot measure.

We publish this process because it is the part a competitor cannot shortcut. None of it is secret method - it is qualification work, and it either has been done or it has not. A tool that has not qualified its judge per dimension, against per-indicator minimal pairs, is guessing about when its model is wrong. We measured it before shipping, and we re-run the exam whenever the model landscape moves, because the best judge for a dimension this year may not be the best judge next year.

3. The weights come from external standards, not from us

A security dimension is formative: its indicators are independent causes of insecurity, not interchangeable symptoms of one hidden trait. That has a hard consequence. You cannot legitimately read the weights off your own data’s internal correlations, because there is no single latent thing to correlate against. So we do not. Each security indicator’s weight is read off its SonarQube rule severity and corroborated by the mean NVD CVSS score of its vulnerability class, computed from a corpus of real CVEs scored by human analysts.

These are two independent scoring bodies, set by different people for different purposes. They converge at a moderate rank correlation (Spearman rho about 0.69 across the vulnerability classes we could map). That convergence is what validates the tier ranking. We are careful not to over-read it: a moderate correlation validates the order, not the exact spacing, and the places where the two standards disagree are themselves evidence that they are genuinely independent rather than two copies of one opinion. We set nothing by hand that a recognised standard did not back.

4. Security fails closed

Security is non-compensatory. A single detected critical defect - a SQL-injection sink, a cross-site-scripting sink, a hardcoded secret, disabled TLS verification - caps the security score regardless of how much good practice surrounds it. One live remote-code-execution path is not averaged away by a wall of validation helpers, exactly as a real security review would not wave it through. So a good security score means “no critical defect was detected, and the protections that apply are present”, bounded by what the instrument can detect. It is not a certificate that the code is secure.

5. We attacked our own weights, and fixed what broke

Weighting by hand invites motivated reasoning, so before shipping we ran the whole scheme through a 40-agent adversarial review: a per-indicator sceptic trying to refute each weight, plus a “reviewer two” panel. It found audit-fatal defects in our first pass, and we fixed every one:

• Fabricated or misattributed rule IDs - a rule cited for the wrong indicator, an invented identifier, a rule reused across two indicators. Fix: only genuinely verified rules keep an identifier; every other weight is honestly labelled as anchored to a named standard rather than a specific rule. No invented anchor survived.
• A standard contradicting a weight by a full severity tier. Fix: conflicts between the two standards are now resolved by one pre-registered rule applied the same way every time, not picked row by row.
• A cross-site-scripting sink weighted too low to trip the gate - a real XSS could have been averaged away. Fix: injection-class violations are floored so a detected sink always fails the score closed.

The panel’s recorded verdict was deliberately not a clean bill of health: the method and the violation pole are defensible; the capability pole is provisional; and a formal procurement audit would still require the remaining provenance and field-validation items to close. We would rather show you that seam than hide it. A scheme that catches its own errors is more trustworthy than one that never reports any.

6. What is confirmed, and what is still maturing

The measurement layer is the strongest part. The security judge (a cheap, open-weight model, roughly a fraction of a cent per call) clears the full reliability gate for this dimension - schema compliance, accuracy, minority-recall and test-retest all around 0.97 - and it is provably sensitive to disguised vulnerabilities without false-alarming on safe-but-suspicious code (for example, MD5 used as a cache key, or randomness used for jitter). Across the whole instrument, re-running the same code gives the same profile (test-retest about 0.94 on real code), and we freeze the result per content hash so every user sees an identical read.

On the scoring model, the clean result is on the violation pole: on crafted, single-issue items the severity-weighted ordering of violations matches the independent CVSS ranking exactly, where a plain equal-count does not. That is the concrete case where weighting demonstrably beats counting.

We are equally plain about the limits. Every security weight is flagged in the product as confirmed, provisional or contested, and we surface those flags rather than present a falsely uniform number. The capability pole’s exact magnitudes are provisional and await human elicitation. On messy real-world CVE corpora the “weighting beats counting” result is not yet significant at scale: the binding constraint there is the judge detecting violations in real (as opposed to crafted) code, not the scoring model, and we do not launder the crafted-item win into a field claim. Severity is a ranking device, not a statement that one issue is a precise multiple of another.

7. Where human expertise enters

Two things are true at once. The weights already encode expert judgement at scale, because SonarQube severities are set by security engineers and NVD CVSS scores are assigned by human analysts, so the scores are anchored to codified professional judgement rather than a model’s opinion. And direct alignment with practising engineers is the next step, not a finished one: we are working toward eliciting senior engineers’ own rankings of which indicators matter most, with inter-rater reliability, and rank-correlating those against our standard-derived weights. High alignment would mean we have codified what good engineers know but cannot easily articulate. That needs a recruited panel of people, not more compute, and we advertise it as in progress.

8. How this template extends to the other thirteen

Security is the template, and the template adapts to whatever external anchor a dimension has:

• Dimensions with a recognised standard (robustness and concurrency are the nearest, and like security they are weakest-link, where one failure is catastrophic) warrant the same external-weight derivation and the same adversarial review, and that work is planned. Until the derivation and its review land, both ship equal-weight.
• Dimensions with a human-rated public dataset(readability and maintainability have direct human-rated corpora) derive weights by regressing the human rating on the indicator vector instead of off a severity catalogue. Honest current result: on real code both saturate (real code is “good enough” almost everywhere), so they ship equal-weight with the gap stated rather than a fabricated weighting.
• Dimensions with no external severity standard (performance, memory, portability, prototype economy, testability, observability, auditability, API ergonomics and resource cost) ship equal-weight by design. Our bar for shipping a weighting is an external standard, and no recognised severity standard exists for these nine constructs, so equal weighting is the defensible choice, not an unfinished one. If a usable external criterion emerges for any of them, it gets the same derivation and adversarial review; until then we do not invent one.

The scoring geometry is discovered per dimension, not assumed: weakest-link with a gate where one failure is catastrophic (security, robustness, concurrency), additive where a straight share-present is genuinely adequate, or a few load-bearing indicators over a hygiene tail. The instrument’s job is to get each dimension’s geometry right, then weight and gate the stable load-bearing few and disclose the rest as hygiene, never to force a flat percentage everywhere.

9. The work does not stop here

What you are reading is a snapshot of an ongoing commitment, not a finished product. The 122 indicators, the 40-agent adversarial review, the 0.938 test-retest, the external-weight derivation and the fail-closed gate are real work that has already been done, and they are not going anywhere. What is also real is the work that has not finished yet: human-expert alignment, field validation on corpora large enough to separate signal from noise, and the extension of the same external-weight treatment to the remaining dimensions where a usable severity anchor exists. Every one of those items is tracked, advertised and in progress.

That is the standard we are holding ourselves to, and it is a higher standard than most AI-powered insights providers set for themselves. Science does not stop at the first result that looks good enough to ship. It keeps asking whether the instrument measures what it claims, whether the weights are defensible, whether the score would survive a sceptical engineer reading the method. We are building Facet to that standard because a profile you can stand behind is worth more than a profile that merely looks confident. There is always more work to do when the goal is a product you can trust, and we intend to keep doing it.

Because Security is the dimension we narrate in full, it is also the one dimension you do not need a subscription to see. A signed-in account gets an essential security review, Security (D6) only, free forever within a capped daily allowance - no credit card, no expiring trial. Run one at /security-review, or get an instant, unauthenticated, zero-model pass over a fixed public checklist of textbook issues (hardcoded secrets, eval on untrusted input, disabled TLS verification, SQL built by string concatenation, shell injection, insecure deserialisation) with no signup at all. The quick pass is not the D6 judge and does not claim to be; it is a smaller, honestly-scoped sibling that exists so an obvious problem is never gated behind anything.

Back to methodology  ·  Data and privacy  ·  Free security review