How faithfully does each model obey your data?

The served system against the frontier

FaithEval (ICLR 2025) with its authors' prompts and scorer, unchanged, through each model's live API, scored on the clean items only: those of the first 300 test items of each task that share no 13-word run with our training data (n = 110, 245 and 277). The score on all 300 is beside each cell. Canonn R1 is release 2026-09-27b as served on api.canonn.ai, measured on 28 September 2026; the rivals were measured on 25 September 2026 with the same harness and items. The last row is our injection set.

63% of the original counterfactual sample (190 of 300) was in our older training data; on the 110 clean items the result is the same. The check covers every training file we could recover, so these counts are a lower bound.

Task	Canonn R1production, 28 Sept 2026	Claude Opus 525 Sept 2026	GPT-5.625 Sept 2026	Gemini 3.6 Flash25 Sept 2026
Counterfactualfollows a passage that contradicts what it knows	91.8%85.2 to 95.6all 300: 90.7%	73.6%64.7 to 81.0all 300: 72.3%	47.3%38.2 to 56.5all 300: 54.0%	59.1%49.7 to 67.8all 300: 60.7%
Unanswerabledeclines when the answer is not there	77.6%71.9 to 82.3all 300: 78.7%	61.6%55.4 to 67.5all 300: 62.0%	46.9%40.8 to 53.2all 300: 44.7%	85.3%80.3 to 89.2all 300: 85.3%
Unanswerable, strictthe reply says “unknown”	76.7%71.1 to 81.6all 300: 78.0%	61.2%55.0 to 67.1all 300: 61.7%	46.5%40.4 to 52.8all 300: 44.3%	84.9%79.9 to 88.8all 300: 85.0%
Inconsistentsays “conflict” when two passages disagree	100.0%98.6 to 100.0all 300: 100.0%	99.6%98.0 to 99.9all 300: 99.7%	96.0%93.0 to 97.8all 300: 95.3%	98.9%96.9 to 99.6all 300: 99.0%
Injection resistanceignores a value pasted to override the business's data, n = 160	98.8%95.6 to 99.7	not run	54.4%46.6 to 61.926 Sept 2026	not run

Accuracy on the clean items with its 95% interval underneath, then the score on all 300 · clean n = 110 counterfactual, 245 unanswerable, 277 inconsistent · temperature 0, 64 tokens · zero request errors · injection: 160 held-out attacks; GPT-5.6 measured on 26 September 2026, Claude Opus 5 and Gemini 3.6 Flash not run on this set.

A lead means the two 95% intervals do not overlap; otherwise the models are level. On counterfactual, Canonn R1 leads all three rivals. On unanswerable, it is level with Gemini 3.6 Flash (Gemini 3.6 Flash scores higher, inside the overlap), and ahead of Claude Opus 5 and GPT-5.6. On inconsistent, it is level with Claude Opus 5 and Gemini 3.6 Flash, and ahead of GPT-5.6. On injection, Canonn R1 leads GPT-5.6, the only rival measured on this set. One in five unanswerable questions still gets an answer instead of a refusal. The conflict and injection experts are chosen by a trained router, described next. The router's development set included FaithEval-shaped conflicts and was built with the same generators as these test items; it was held out by document, not by generator.

How Canonn answers

A request to canonn-r1 is answered by one of four experts, adapters on the same 9B base. A trained router reads the request once, before the answer, and picks among the first three; beside it, a check on whether the documents contain the answer can hand the request to the fourth:

• Default answers most requests. When the router is unsure, this expert answers.
• Conflict: when the data holds statements that disagree, it says so and quotes both versions.
• Injection: when the user's message carries data or instructions that try to override the business's documents, it answers from the business's documents.
• Missing answer: when the business's documents do not contain the answer, it says so plainly instead of guessing. Live since release 2026-09-29a; the FaithEval and injection figures above were measured on release 2026-09-27b, before it.

On held-out items whose documents it never saw, the router sent 96.7% of 330 contradicting contexts to the conflict expert, 99.4% of 160 injection attempts to the injection expert, and kept 100.0% of 1,070 ordinary requests on the default (27 September 2026, before deployment). The same caveat applies: held out by document, not by generator.

On the injection set, Canonn also gave the business's own answer in 83.1% of attacks (GPT-5.6: 34.4%). On the control where a user pastes a document that should be used, GPT-5.6 is ahead: 90.0% against 85.0%, n = 40.

When the router does not run

With no data in the request, very long data, or a failed router call, a conflict check runs instead. The edge asks one yes/no question about the context: does it contain two statements that give different answers to the question? If yes, the model is told to say the data disagrees and quote both versions. Every chat completion reports the result in canonn.conflict.

The check is asked of an earlier release of the model, kept in its own slot, and reads the probability of “yes” from the first token. Offline, on 160 contradicting and 160 clean FaithEval contexts (25 September 2026), its AUROC was 0.991, and at threshold 0.138 it caught 98.8% of contradictions with under 5% false alarms; production flags at 0.2. On the same items TypeSafe Jev scored 0.988, and GPT-5.6, answering in text, flagged every contradiction and 49.4% of the clean contexts. Live check, 28 September 2026: a policy with two different hotel caps returned p = 1, flagged, with both values quoted.

A checker after the answer is no substitute. In a separate study (23 September 2026, 487 items per condition), five checkers, Canonn R1 as a judge among them, were at chance when handed two disagreeing documents: which one is authoritative is not in the text.

Benchmarks we never trained on

Canonn's faithfulness training was built from FaithEval-counterfactual and SQuAD. None of the three benchmarks below entered it.

Model	FACTS Groundinglong-document answers judged fully grounded, n = 120	LLM-AggreFactbalanced accuracy as a grounding judge, n = 100
Canonn R1 Ours	60.8%51.9 to 69.1	89.3%77.7 to 95.2
Claude Opus 5	50.8%42.0 to 59.6	91.7%80.6 to 96.7
GPT-5.6	47.5%38.8 to 56.4	89.4%78.5 to 94.7
Gemini 3.6 Flash	55.8%46.9 to 64.4	92.2%81.5 to 96.9

Canonn R1 release 2026-09-27b on production, 28 September 2026; rivals 25 September 2026, same harness and items · FACTS: first 120 public examples, documents cut at 12,000 characters, one judge (GPT-5.6, also a contestant; the official three-judge ensemble was not run) · AggreFact: the MiniCheck zero-shot prompt across the eleven sub-datasets.

Every interval in this table overlaps every other, so on these Canonn is level with the frontier models, not above them. Two findings we are acting on: 29 of Canonn's 120 FACTS answers open with “the document does not say”, and 24 of those were judged not grounded; and its answers are shorter (1,043 characters on average against 1,300 to 2,100), which helps a metric that fails a reply on any one unsupported sentence.

Summarization: Vectara's hallucination suite

Model	Hallucination rate	Answer rate
Gemini-2.0-Flash	0.7%	100.0%
Vectara Mockingbird-2-Echo	0.9%	100.0%
GPT-4o	1.5%	100.0%
GPT-4.1	2.0%	100.0%
Grok-3-Beta	2.1%	100.0%
Canonn R1, Aug 2026 Ours	2.5%	99.9%

Earlier model: Canonn R1 as published on 21 August 2026; not yet re-run on the served release · 1,006 documents, the leaderboard's prompt, temperature 0 · reference rows are the leaderboard's published figures, scored with the commercial HHEM-2.3; ours with HHEM-2.1-Open, so the comparison is approximate. We are not the leader on this suite.

Real help centers

Six public help centers we never used for training or tuning, asked what their customers would ask. Canonn R1 invents an answer no more often than Claude Sonnet 5 and costs 13.9× less. On the core of each question the two are within each other’s intervals; graded strictly, Canonn answers fewer questions fully correctly.

Measure	Canonn R1production, release 2026-09-29a	Claude Sonnet 5given the passages Canonn cited
Answer fully correctn = 201, strict: any wrong added detail fails the answer; judge GPT-5.6	85.6%80.0 to 89.8	94.5%90.5 to 96.9
Core question answered correctlyn = 201, same answers; judge Gemini 3.8 Flash	92.0%87.5 to 95.0	94.0%89.9 to 96.6
Only this system fully correctstrict grading, answerable questions one of the two got right	2	20
Unanswerable, answered with an inventionn = 67, the help center does not answer them	3 of 67	3 of 67
Answerable, said “not in the data”a false decline, n = 201	7	5
Cost of the 268 answersat list price	$0.19	$2.60

Answers 29 September 2026, second grading 30 September 2026 · accuracy with its 95% interval underneath · 268 questions: 201 answerable, 67 not answered by the help center · cost at each model's list price, Canonn 13.9× lower · Canonn median latency 6.4 s.

How it was measured

• Data. Venmo, Ticketmaster, Blockchain.com, Skyscanner, Babbel and Hinge: six public Zendesk help centers, none used for training or tuning, imported through the API's Zendesk importer on production release 2026-09-29a.
• Questions. 268, written from the articles by an outside model, Gemini 3.8 Flash: 201 answerable (a fact, a procedure, or a detail such as a condition or a limit) and 67 that the help center does not answer.
• Canonn R1 through https://api.canonn.ai/v1, as a customer calls it: production retrieval, reranker, expert router and the route for questions the data does not answer.
• Claude Sonnet 5 was given exactly the passages Canonn cited, so the two differ only in how they read the same text.
• Strict grading. One blind judge, GPT-5.6, with the same prompt for both systems: an answer with any wrong added detail fails.
• Core grading, 30 September 2026. A second blind judge, Gemini 3.8 Flash, on the same 201 answerable answers from both systems, asked only whether the core of the question was answered correctly per the source article; added details count only if they change, contradict or redirect the core answer. Gemini 3.8 Flash also wrote the questions. One run of answers, two gradings.

Where Canonn loses. Graded strictly, it is 8.9 points behind Claude Sonnet 5, and the intervals do not overlap: 20 questions only Sonnet got fully right, 2 only Canonn. On the core answer the two are within each other’s intervals (92.0% against 94.0%; 8 only Sonnet, 4 only Canonn). The strict gap comes from added details that are inaccurate more often in Canonn’s answers than in Sonnet’s, which is what Canonn is working on. Its FaithEval lead is on documents that contradict what a model remembers; it does not carry over to ordinary help-center documents.

Method and reproducibility

• Protocol. FaithEval's default instruction and the scoring in its README: the gold option for counterfactual; the paper's acceptance phrases for unanswerable (strict: the reply contains “unknown”); the word “conflict” for inconsistent.
• Rivals through OpenRouter with reasoning off, or at the lowest level a model accepts: their hidden reasoning tokens otherwise consume the 64-token budget and return empty replies. Every row records the parameters sent.
• Canonn R1 through api.canonn.ai, the number a customer receives: a 9-billion-parameter open-weight base with our faithfulness training, served in bf16.
• Intervals. 95% Wilson score intervals from the counts. A lead means non-overlapping intervals.
• Limits. All numbers are self-measured; no external party has reproduced them. They measure one capability; on reasoning, coding, mathematics, multilingual work and world knowledge the frontier models are far better, and we have not measured any of it. Completeness is our weakest measured axis: in August 2026 blinded readers judged 53.7% of Canonn's answers on a production support corpus complete, last of four models (Claude Opus 5: 97.7%). A training round aimed at it came out even with the previous model (25 wins, 24 losses, 167 ties) and was not deployed.