Known. Computed. Defended.
Three conditions applied to every task, a verifier with no judge model in it, and an audit that attacks the verifier before anyone else can. This page is the whole method, including the parts that are inconvenient.
Only what existed on the task date.
A record is available to the agent if, and only if, it was published at or before the task's as-of time.
What "available" means
Every record in the fabric carries three stamps: when it was published, the period it is effective for, and the hash of the stored source. The harness applies one rule: published_at <= as_of. A filing published at 04:12 UTC is available to a task dated that midnight only if the task says so; a restatement published four months later is never available to a task dated before it, however much it "corrects" the original. The agent is scored against the figure the market had, not the figure the market later learned.
This is the operational meaning of point-in-time, and it is the part most finance training data skips. Restatements are not rare — 479 US filers restated in 2024 — but the larger leak is simpler: a quarter's figures are not public until the filing lands, a median 28 days after the quarter ends, and a dataset stamped by period rather than by publication hands them to the model a month early. Quantitative investors have paid for point-in-time databases for exactly this reason since the 1990s; the sell side calls look-ahead "probably the most common bias in backtesting" (Deutsche Bank, 2014). Training sets for language models are rarely built that way. The reading list has the literature that measures what the leak is worth.
The agent's own memory
The harness controls the documents. It cannot control what a model read during pretraining, and that memory is now measured rather than suspected: GPT-4o recalls S&P 500 daily closes from before its training cutoff with 0.61% error, and from after it with 16.87%. Instructing the model to behave as if it were an earlier date does not close the gap; masking the company's name does not either — it identified Apple from anonymised earnings calls every time (Lopez-Lira, Tang & Zhu, 2025). Three things are done about this, and none of them is pretending it is not a problem.
- Tasks are scored on quantities memory cannot produce. The figure on a given page of a given filing. The exact cell a change touches, and by how much. The recorded assumption and the threshold set against it. Recollection of the stock's path does not yield any of these; only reading the available documents and computing does.
- Where a family is sensitive to memory, the gap is measured and reported. The same family is run on issuers and periods past the model's stated training cutoff, and the difference is printed in the audit, not absorbed into the headline number.
- Trajectory data makes the point moot. An expert working a task inside the terminal in 2026 on documents stamped 2024 produces a trace of the work, not a guess at the outcome. What is learned from it is the procedure.
What the agent is told
A task statement names the as-of time, the universe it may read (issuers, document kinds, series), the question, and the answer type with its fields. It is told which tools it has — document retrieval over the available set, the compute engine, and nothing else — and that reads of anything later than as-of are logged and fail the task. It is not told the answer's magnitude, its sign, or which document holds it.
Abstention is an answer
Some tasks cannot be answered from the available record — the figure is not yet published at as-of, the assumption was never recorded — and some are seeded that way on purpose. The agent may say so, and is scored on whether that was true. Abstention is counted separately from error, as it is on Quantify Terminal's own accuracy page, because a high hit rate with a hidden abstention rate is not accuracy. The option also matters for training: on ImpossibleBench, giving a model an explicit way to abort cut its rate of gaming unsatisfiable tasks from 54–76% to 9% (2025).
Arithmetic from an engine, not an opinion.
Every number the verifier checks against was produced by a deterministic engine from the stamped inputs, and can be reproduced from them.
Ground truth for every numeric field is computed by the same model compiler that runs inside Quantify Terminal: ratios, growth rates, trailing figures, every cell of an estimate model, and the deltas a new filing produces in them. The engine is deterministic, the inputs are the available records and nothing else, and the run that produced a task's truth is stored with the task. In the sample pack the engine's outputs ship beside the inputs so the truth can be recomputed outside our systems.
The check is numeric, to a tolerance stated on the task. A submission within tolerance passes; one outside it fails; there is no "close". Tolerances are set by the author from the precision the source supports, and are visible in the record, so a buyer can tighten them and re-score.
Why there is no judge model
The alternative — asking a language model whether the agent's answer looks right — is the industry's default, and it is measurably unreliable. On JudgeBench, GPT-4o as a judge agreed with the verifiable answer 56.6% of the time, against 50% for a coin (2024); on AgentJudgeBench, six LLM judges of agent output plateau at 77–82% alignment with human verdicts on hard tasks regardless of scale (2026); and agent-written verifiers reject invalid solutions more reliably than they accept valid ones (VerifierBench, 2026). A judge model also makes the audit meaningless, because its error rate changes with every prompt and every model update. A deterministic verifier has one error rate, it can be measured, and once measured it stays measured until the verifier version changes.
Where a task has a genuinely free-text component — the explanation of a reconciliation break, say — it is scored on the structured claims inside it: which sources it names, which figures it cites, whether they resolve. The prose itself is not graded.
Sources, assumptions, thresholds — linked, not described.
An answer is defended when every claim in it points at something on the record that supports it.
Three link checks, each mechanical. The source span the agent cites must resolve: to a stored document, by hash, at the page or location given, containing the figure claimed. The assumption it names must be the one on record for that thesis at that date, by identifier, not a paraphrase that fits. The threshold it cites must be the one that was set, not one that makes the answer work.
Credit is per condition. An agent that reads only what it should, computes the right delta, cites a span that resolves, and names the wrong assumption scores 0.75 and a note saying which it missed. Partial credit is withheld, never rounded up, and the components are reported separately so a training signal can weight them. A right number with no defence is worth exactly what it is worth to a portfolio manager: a number someone else has to go and check.
The verifier is attacked before it ships, and the damage is printed.
Robustness to reward hacking is the first thing a lab asks of an environment. The audit is the answer, with numbers.
A scoring function is a target. One frontier model, evaluated by METR, reward-hacked 39 of 128 runs on a research-engineering benchmark — 21 of 21 on one task — and telling it not to cheat had no effect (2025). Even a human-validated benchmark leaks: an audit of SWE-bench Verified found 28.5% of a sample would accept a wrong patch (2026). Any environment that has not been attacked by someone trying to do exactly that is an environment whose numbers mean less than they appear to. So before a pack ships, it goes through what the literature calls an adversarial verifier audit.
- Authoring. A contracted practitioner writes the task, the answer and the rubric from the available documents, and states the tolerance.
- Second review. A second practitioner works the task blind. Disagreements are recorded with both positions, resolved by a third, or the task is withdrawn.
- Ground truth. The engine computes every numeric field from the stamped inputs; the run is stored with the task.
- The attack pass. A team whose job is to score without working: guess from priors; read past as-of; exploit units, signs, rounding and tolerance; cite a span that exists but does not support the claim; satisfy the rubric's words without its meaning; game the answer format. Every success is logged against the task and the verifier version.
- The fix pass. Every hole is closed — in the verifier, in the task, or in the rubric — and the attack pass is re-run against the fix.
- Measurement. The verifier's false-positive and false-negative rates are measured on the dev set (definitions below), against adversarial submissions and against held-out expert submissions, and written into the audit report with their denominators.
- Freeze and version. The verifier is versioned; every score names its version. Held-out sets are given a retirement date and rotated.
Definitions
- verifier false positive · soundness
- A submission the verifier credited, in whole or in part, that a reviewing expert judges did not meet the condition credited — the probability of reward given an incorrect answer. Measured per condition — known, computed, defended — on the adversarial set and on the expert-submission set.
- verifier false negative · completeness
- A submission the verifier failed, in whole or in part, that a reviewing expert judges did meet the condition failed — the probability of no reward given a correct answer. Measured the same way. Over-rejection is a known failure of generated verifiers and is reported, not hidden in a higher pass bar.
- attack pass result
- For each attack class: attempts, successes before the fix pass, successes after. Reported per task family.
- memory gap
- For families sensitive to the model's own training data: the score difference between issuers and periods before and after the model's stated cutoff.
The literature this stands on.
The leak, the benchmarks that measure what is left, and the evidence that scoring needs auditing.
The leak
- 2025The Memorization Problem: Can We Trust LLMs' Economic Forecasts?Lopez-Lira, Tang & Zhu. GPT-4o recalls S&P 500 daily closes before its cutoff at 0.61% error and after it at 16.87%; date instructions and entity masking do not close the gap.
- 2026DatedGPTYan et al. Twelve models trained with strict annual cutoffs; look-ahead-biased models earn a spurious premium of 26.4 bps per standard deviation.
- 2025Chronologically Consistent Large Language ModelsHe, Lv, Manela & Wu. Models trained only on text available at each date. The return-prediction lift from leakage is modest — which is the point: point-in-time is about the validity of the signal, not inflated alpha.
- 2023Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment AnalysisGlasserman & Lin. Anonymised headlines beat the originals in-sample; memorised names distract as much as they leak.arXiv 2309.17322 · Journal of Financial Data Science, 2024
- 2018Forecasting through the Rearview Mirror: Data Revisions and Bond Return PredictabilityGhysels, Horan & Moench. Real-time data substantially reduces the predictive power of macro variables.
- 2014Seven Sins of Quantitative InvestingDeutsche Bank Quantitative Strategy. Look-ahead is "probably the most common bias in backtesting"; restatement bias; the case for point-in-time databases.
What is left when the leak is closed
- 2025FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language ModelsAfterQuery. o1 48.7% overall and 2.2% on assumption-based questions; GPT-4o 39.2%.
- 2026Finance Agent Benchmark v2.1Vals AI. 449 private tasks; best overall 67.85%, best on financial modelling 34.52%; v1 retired as saturated.vals.ai · 29 Oct 2026
- 2026DAYJOBSurge AI. 80 finance tasks averaging 16.6 human-hours and a median 57.5 pass criteria each, built for held-out evaluation; top model 23.9%.
- 2023FinanceBenchPatronus AI. GPT-4-Turbo with retrieval refused 68% and answered 13% wrong on the human-evaluated sample.
- 2024
- 2025FailSafeQAWriter. The most robust model still fabricated information in 41% of perturbed cases.
Scoring needs auditing
- 2025Recent frontier models are reward hackingMETR. o3 reward-hacked 39 of 128 RE-Bench runs and 21 of 21 on one task; "do not cheat" instructions had no effect.
- 2025ImpossibleBenchGPT-5 gamed 54–76% of unsatisfiable tasks; an explicit abort option cut it to 9%.
- 2026An audit of SWE-bench VerifiedRajan. 28.5% of a sample accepts wrong patches; pass@1 is 14.1 points higher on hackable tasks than on robust ones of the same difficulty.
- 2024
- 2026AgentJudgeBenchSix LLM judges of agent output plateau at 77–82% alignment with human verdicts on hard tasks, regardless of scale.
- 2026VerifierBenchAgent-written verifiers reject invalid solutions more reliably than they accept valid ones.
- 2026The state of RL environmentsEpoch AI, from interviews with environment builders and buyers. Robustness to reward hacking is the labs' first selection criterion.
- 2026Verity-RedTeamOpen-source adversarial auditing of RL-environment verifiers. Defines soundness as the probability of reward given an incorrect answer and completeness as the probability of no reward given a correct one — the two rates every Labs audit reports.