The sample pack.
Enough of the v0 dev set to judge the thing itself: the tasks, the documents they stand on, what the engine computed, what the verifier said, and how the audit is written up. Sent to teams who would use it, by email, usually within two working days.
Seven things, each of which you can check.
Nothing in the pack is a slide. Every item is a file you can open, diff against the schema, or run the engine's numbers against.
Tasks from each of the four families.
Quant research, analyst modelling, reconciliation and thesis break, in the record format shown on the home page: task statement, as-of, universe, answer type, author and reviewer identifiers.
The stamped documents each task stands on.
Filings and disclosures with published and effective timestamps and source hashes, including the ones dated after T, so you can see what the harness refuses and why.
What the deterministic engine computed.
Every figure the verifier compares against, with the inputs it was computed from, so the ground truth can be reproduced outside our systems.
Scored submissions, right and wrong.
Verifier output for correct, partially correct and adversarial submissions on each sample task — including the ones the attack pass tried.
How the audit is written up.
The report template with its definitions: what counts as a verifier false positive, what counts as a false negative, how each is measured on the dev set, and how the attack and fix passes are recorded.
The record schema, documented.
Field names, types and constraints for tasks, documents, trajectories and verifier output, with a dataset card in the Datasheets-for-Datasets format and Croissant metadata. Field names are final; the schema below is the current draft.
A draft of the licence.
What the pack grants, what it withholds, how per-record rights profiles are expressed, and the terms under which held-out sets are delivered.
One task record, in the draft schema.
JSONL, one record per line; the same fields in Parquet. Timestamps are UTC. Hashes are SHA-256 of the stored source. Values here are illustrative.
// task record · qtl-v0 · draft schema · values illustrative { "task_id": "qtl-v0-tb-0147", "family": "thesis_break", "as_of": "2024-02-15T00:00:00Z", "universe": { "issuers": ["…"], "documents": 9, "series": 2 }, "known": { "available": 6, "refused": 3, "rule": "published_at <= as_of" }, "question": "Which recorded assumption did the FY2023 10-K break, by how much, and against which threshold?", "answer_type": { "assumption_id": "string", "delta_pct": "number", "threshold": "number", "source_span": "span" }, "compute": { "engine": "qt-compiler", "cells": 14, "tolerance": 0.05 }, "rights": { "profile": "train+eval", "redistribute_source": false }, "provenance": { "author": "exp-0412", "reviewer": "exp-0077", "disagreements": 0 }, "documents": [ { "doc_id": "…", "kind": "10-K", "published_at": "2024-02-02T04:12:00Z", "effective": "FY2023", "sha256": "54ab…6fc5c", "available": true }, { "doc_id": "…", "kind": "10-K/A", "published_at": "2024-06-12T20:15:00Z", "effective": "FY2023", "sha256": "c1e9…02a7d", "available": false } ] } // verifier output for one submission { "task_id": "qtl-v0-tb-0147", "submission_id": "…", "known": { "pass": true, "reads_after_as_of": 0 }, "computed": { "pass": true, "submitted": -1.77, "engine": -1.77, "tolerance": 0.05 }, "source": { "pass": true, "resolves": true, "sha256": "54ab…6fc5c", "page": 41 }, "defended": { "pass": false, "reason": "threshold cited; assumption record not linked" }, "score": 0.75, "verifier": { "version": "v0.3", "deterministic": true } }
Tell us what you are training or evaluating.
Six questions. The form writes an email in your own mail client to data@labs.quantifyterminal.com; nothing is sent from this page. If you would rather just write, write.