Labelling projects fail in three ways: labels that break the schema, annotators who quietly disagree, and a batch that looks finished but is wrong. This tool answers all three with numbers.

Validation — the label set, the minimum confidence and whether spans are required come from a schema file, not from the code. A record is rejected, with its file, line and one exact reason, when the label is outside the schema, the confidence is out of range or below the agreed floor, a span is reversed or runs past the end of the text, a required field is missing, the line is not valid JSON, or one annotator labelled the same item twice. A malformed line never stops the run and never disappears.

Agreement — raw match rate flatters an unevenly distributed label set, so the report gives both it and Cohen’s kappa, which subtracts the agreement expected by chance. On the sample: 0.80 match, 0.7436 kappa, with each of the two disagreements listed by item so a reviewer sees the actual conflicts. Kappa is reported as undefined, never as a perfect score, where it is undefined.

Against a gold set — accuracy per annotator plus per-label precision, recall and F1, which is what shows whether one specific label is the weak point. A gold file that fails its own schema is refused outright: a number computed against a broken reference looks like quality and is not. --min-kappa makes the agreed bar a build gate, so the run exits non-zero when a pair falls below it.

Python 3.9+, standard library only, 20 output-level tests. Code, fixtures and sample report: https://www.twine.net/signin

A messy contact export goes in. Three files come out: the rows that are provably correct, the rows that are not with the exact reason for each, and a JSON report.

Normalised — spacing, email case and shape, international phone digits, dates (2026-01-15, 15.01.2026, unambiguous 25/12/2026) into ISO, amounts written as 1 234,56 or 1,234.50, and duplicates on email, phone or id.

Refused rather than guessed — 03/04/2026 can be 3 April or 4 March, so that row is rejected with that reason instead of resolved silently. The same goes for an impossible date such as 2026-02-30, an invalid address, a phone outside 8-15 digits, and a negative or non-numeric amount. Every rejected row keeps its source line and one sentence explaining the decision, so the list can be corrected at source and the run repeated.

On the sample data — 12 rows in, 4 clean, 8 rejected with stated reasons. Output files are written through a temporary file and renamed, an existing file is never overwritten without an explicit flag, and a missing column, an empty file or a missing output directory stops the run before anything is written.

Python 3.9+, standard library only, 20 output-level tests. Code, fixtures and sample output: https://www.twine.net/signin