Point-in-time training environments, expert workflow traces, and audited evaluation — for the companies teaching AI to work in finance.
A model that has read the restated filing, the next quarter and the price path cannot be scored on whether it would have got there on its own. The literature measures the leak; the benchmarks measure what is left when it is closed.
GPT-4o's error recalling S&P 500 daily closes before its training cutoff, and after it. Telling the model to ignore later knowledge does not close the gap; masking the company does not either.
Lopez-Lira, Tang & Zhu, The Memorization Problem · arXiv 2504.14765 · 2025
of realistic financial-analysis tasks fail with frontier models — hand-spreading, accounting conventions, incomplete information.
FinanceQA, AfterQuery · arXiv 2501.18062 · Jan 2025
is the best any model scores on the financial-modelling tasks of the leading independent finance-agent benchmark. Overall, the best model reaches 67.9%.
Vals AI, Finance Agent Benchmark v2.1 · vals.ai · 29 Oct 2026
of one frontier model's runs on a research-engineering benchmark were reward hacks: the score was earned without doing the task. Which is why the verifier is audited before the pack ships.
METR, evaluation of OpenAI o3 · metr.org · 2025
Three products on the same point-in-time fabric. In each, an agent earns credit only for what it could have known, computed and defended — and each pack carries the audit that says how well the scoring held.
Documents, fundamentals, estimates and prices, each stamped with the moment it became available. Nothing later than the task date enters the context — not the restated figure, not the next quarter, not the price path.
Contracted CAs, CFAs and former analysts working authored tasks inside Quantify Terminal and Excel. Screen, keystrokes, the datapoint each number came from, timestamps, and the rubric the task was scored on.
Tasks in which the agent sees only what was public at date T. Thesis-break tasks, event streams, exact numeric checks, and a verifier that was attacked before it shipped.
Every pack ships with an adversarial verifier audit and measured false-positive and false-negative rates. We do not print numbers we have not run. How the audit works →
One task from the thesis-break family, in the format every pack uses. Field names are final. Values are illustrative until the v0 dev set publishes.
Three conditions, applied to every task. An agent that meets two of them gets partial credit and a note saying which it missed. The full method, including the audit protocol →
The agent may use only what existed on the task date. Each record carries its own availability timestamp, and the harness refuses anything later — including the restated figure.
Tasks are scored on quantities a model cannot have memorised: the exact figure in the filing, the exact cell the change touches. Knowing what happened next does not produce them.
Arithmetic comes from a deterministic engine, never from a model's opinion of its own numbers. Every ratio, growth rate and model cell is recomputed from the stamped inputs.
A correct answer is a reproducible one. The check is numeric, to a stated tolerance, against the engine's figure — not a judge model's impression of it.
Each conclusion must link to its sources, assumptions and thresholds. The span has to resolve; the assumption has to be the recorded one; the threshold has to be the one that was set.
If the agent cannot show why, it gets no credit for that part — and partial credit is withheld, not rounded up.
Before a pack ships, one team tries to earn credit without doing the work — guessing from priors, reading past T, gaming units and rounding, fabricating a source. Another closes every hole they find. The pack ships with both reports and two measured rates: how often the verifier credited a wrong answer, and how often it rejected a right one.
Four families in the v0 dev set, each authored by a practitioner who has done the work for a living and reviewed by a second who has too. Custom families are built on the same fabric.
Form and test a hypothesis using only data stamped available at the task date. Scored against what happened next — which the agent never saw.
Update an estimate model when a new filing lands, and defend each changed cell. The engine recomputes the sheet; the agent has to match it and say why.
Tie figures across filings, data feeds and models. Find the breaks, size them, and explain which source is right and how you know.
Given numeric thresholds and an event stream, find which recorded assumption broke, by how much, and what it means for the position.
Quantify Terminal is a research terminal for investment teams. Its data fabric stores every observation with the moment it became available and the licence it carries; its model compiler recomputes every cell deterministically; its decision ledger records what was believed, on what evidence, before the fact. Labs turns that machinery into training data.
Every record carries all three. The harness reads the first and refuses anything the agent should not yet have. The model never sees a figure early, and never sees the restatement at all.
Display, derived use, AI retrieval, redistribution: the terminal records what each source permits and enforces it at read time. Packs include only what their licences permit for training, and say so per record. How the terminal handles data rights →
Ratios, growth rates, trailing figures, model cells: all recomputed from stamped inputs by the same engine that scores the task. The verifier and the ground truth cannot disagree with each other.
Theses, assumptions and thresholds are recorded with the date they were set. That record is the ground truth for thesis-break tasks: the agent has to find which recorded belief the new filing broke — not invent one that fits.
Labs buy speed and a reward signal that cannot be gamed. Agent builders buy a held-out test their customers have not seen. Validation teams buy evidence in a form that survives review.
Environments with verifiable rewards and expert trajectories in a domain where the public benchmarks are either saturated or already in the pretraining corpus. Delivered in your harness format, with the audit attached.
Point-in-time, held-out evaluation of an agent before customers run it on live books — the number it would have got, on the day, with only what was public. Private test sets that rotate.
Evidence that an agent's finance answers were time-consistent, computed and sourced — task by task, with the verifier's own error rates stated. Built for the file, not the slide. From v1.
Scale AI says nearly half of its new data projects now involve RL environments (Feb 2026). Anthropic's finance launch cited an independent finance-agent benchmark rather than its own (May 2026). OpenAI's financial-services lead: academic benchmarks "are not interesting to this industry… it's about real-world utility" (Sep 2026). And in one published case, expert-written credit tasks plus a 100-task held-out set took an open model from 47% to 74% perfect solves (AfterQuery with Hebbia, Oct 2026).
The things that end a data business are not copyright claims. They are a client's document in a training set, an insider's knowledge in a task, a test set that leaked. The rules below are how none of those happen. The full account →
Filings, disclosures, prices and documents whose licence profile permits the use. No Quantify Terminal client data, no client research, no private documents — ever, under any anonymisation.
Former and independent practitioners rather than serving analysts. No one authors a task on an issuer they currently cover or hold. Screening is recorded against every episode they produce. Work with us as an expert →
Every trajectory carries its author, its reviewer and the rubric it was scored on. Nothing in a pack is unattributed, and nothing in it can be traced back to a person who did not consent to be there.
Dev-set samples are shared freely. Held-out tasks are not published, are delivered under terms that forbid training on them, and are retired on a schedule, so the number a model gets means what it says.
This page says what exists and what does not. When a number appears here it will come with its denominator.
Sample tasks, stamped documents, verifier output and the audit format are available on request to teams who would use them. Request the sample pack →
Verifier false-positive and false-negative rates measured on the dev set, with the attack pass and the fix pass written up. It publishes with the pack, not before.
Expert sessions recorded inside Quantify Terminal and Excel with datapoint links, and the tool interface by which an agent works a task under the harness.
Each is either someone else's business or a way to end ours.
The harness controls the documents; it cannot control what a model read during pretraining, and that memory is real: GPT-4o recalls S&P 500 daily closes before its training cutoff with 0.61% error and after it with 16.87%, and neither instructing it to ignore later knowledge nor masking the company name closes the gap (Lopez-Lira, Tang & Zhu, 2025). So the tasks are built to make that memory useless. They are scored on exact quantities — the figure on page 41 of this filing, the cell this change touches, the recorded assumption and its threshold — which a model's recollection of what happened to the stock cannot produce. Where a family is sensitive to it, we also test on issuers and periods past the model's stated cutoff and report the gap. The method page sets out the checks and the literature they rest on.
Tasks from each of the four families; the stamped documents each task is built on; the deterministic engine's output for each; verifier output in the format shown above; the audit report format with its definitions of false positive and false negative; the record schema; and a draft of the licence. Full contents →
Contracted chartered accountants, CFA charterholders and former sell-side and buy-side analysts, each conflict-screened and on a cooling-off period from the issuers they author. Every task has a second expert's review; where the two disagree the disagreement is recorded, not averaged away. Author and reviewer identifiers travel with every record.
No. Numeric checks run against a deterministic engine to a stated tolerance. Source checks resolve a cited span to a stored document and its hash. Defence checks test that the assumption and threshold the agent names are the ones on record. There is no judge model in the scoring path, so the same submission scores the same way every time — which is also what makes the audit's error rates meaningful.
A non-exclusive licence to use the pack for training and evaluation, with the rights profile of every record stated so you can see what each one permits. Third-party content is included only where its own licence permits that use; where a source forbids redistribution, the pack carries the derived record and a pointer, not the content. Held-out sets come under terms that forbid training on them. The draft licence is in the sample pack; the final one is negotiated per agreement.
Datasets as Parquet and JSONL with a documented schema. Trajectories as screen video with a synchronised event log and datapoint links, plus the task statement, rubric and gold output. Environments through a tool API against sandboxed documents, with the verifier callable as a service, in the harness format you specify — or published to a private Hugging Face repository under the rl-environment tag in Harbor, verifiers or OpenEnv format.
Yes. Custom families are authored on the same fabric with the same three checks and the same audit. Tell us the workflow and the failure you are seeing; the sample-pack form has a field for it.
Public filings and disclosures, licensed data whose profile permits the use, and expert-created work. Never: Quantify Terminal client data or research, material non-public information, scraped content of unclear rights, or market microstructure. The trust page sets it out in full.