THE AGREEMENT CARD · do the graders agree, and does the judge?
WAKES WHEN someone labels eval data, writes a rubric, or wants to trust an AI judge
THE LOOP
two people grade the same 50 cases, blind
agreement.py kappa · the table, the splits
rewrite the rubric from the disagreements
calibrate · the judge against the people
judge runs the routine; people the rest
NEVER
Trust a rubric nobody has disagreed over
Read kappa without the raw agreement beside it
Let a lenient judge grade the launch
IT MAKES kappa — two graders: agreement, Cohen's kappa, the table, every disagreement; fleiss — many graders: Fleiss' kappa and the items they split on; calibrate — the AI judge against people: agreement, kappa, leniency, the cases to read.
THE STANDARD agreement before accuracy; kappa beside raw agreement, always; low kappa indicts the rubric, not the grader; an AI judge is calibrated against people; a lenient judge inflates every score it touches; under thirty items the figure is rough.
PROOF SELFTEST 10/10 · witnessed in Claude 2026-10-09 · Edition 1.
THE SEAL 98 of 100 · LEGENDARY · WITNESSED · Trigger 5 · Machinery 5 · Law 5 · Portability 5 · Proof 5
IT CAN BE WRONG an AI reading a text file: it can be wrong and can invent a number, a rule, a name or a date; check what matters against your own records and the source; the risk is yours. THE-CLAUSES.md rides in the packet and is part of the licence.
THE PUNCH · SHA-256 OF SKILL.MD · 5A588E01324E4DA6AC4E34D0
CITE THIS CARDTHE AGREEMENT CARD, Edition 1. Lund Studio LLC, Westminster, Colorado. https://lundr.tech/?view=agreement-card. Punch 5A588E01324E4DA6AC4E34D0. Retrieved 2026-10-10.
What it is.
Measures whether graders agree before their labels are trusted: Cohen's kappa for two graders with the agreement table and every disagreement listed, Fleiss' kappa for many graders with the items they split on, and a calibration of an AI judge against human labels that shows how lenient or strict the judge is and which cases to read.
It carries a machine that shows its working: kappa — two graders: agreement, Cohen's kappa, the table, every disagreement; fleiss — many graders: Fleiss' kappa and the items they split on; calibrate — the AI judge against people: agreement, kappa, leniency, the cases to read — 10 checks in its selftest, every command on the record. It will never trust a rubric nobody has disagreed over; never read kappa without the raw agreement beside it; never let a lenient judge grade the launch.
Six laws, written into the card.
They are in the card itself, so the AI follows them.
AGREEMENT BEFORE ACCURACYA label two graders cannot agree on measures nothing; grade fifty cases twice before grading a thousand once.
KAPPA BESIDE RAW AGREEMENT, ALWAYSOne without the other misleads when a label is common.
LOW KAPPA INDICTS THE RUBRIC, NOT THE GRADERRewrite it with the disagreements in front of you.
AN AI JUDGE IS CALIBRATED AGAINST PEOPLEAn AI judge is calibrated against people, on the same cases, before and after every prompt or model change.
A LENIENT JUDGE INFLATES EVERY SCORE IT TOUCHESThe gap is measured in points and written on the report.
UNDER THIRTY ITEMS THE FIGURE IS ROUGHUnder thirty items the figure is rough, and the machine says so.
What rides in the card.
In the card’s own voice, with a machine that shows its working.
THE GUIDEreferences/KAPPA-IN-PLAIN-WORDS.md
THE MACHINEagreement.py · three commandskappa — two graders: agreement, Cohen's kappa, the table, every disagreement; fleiss — many graders: Fleiss' kappa and the items they split on; calibrate — the AI judge against people: agreement, kappa, leniency, the cases to read. Every command prints its working and refuses a bad input; 10 checks in its selftest, every command on the record.
What it does once your AI has it.
kappa Two graders: agreement, Cohen's kappa, the table, every disagreement
fleiss Many graders: Fleiss' kappa and the items they split on
calibrate The AI judge against people: agreement, kappa, leniency, the cases to read
On the record Two graders on fifty cases — the textbook table, kappa 0.40, every disagreement listed; three graders on four items worked by hand to kappa 1/3; and an AI judge that passed everything, measured lenient by 30 points with its fifteen false passes named.
What it will never do.
It will never trust a rubric nobody has disagreed over; never read kappa without the raw agreement beside it; never let a lenient judge grade the launch.
Kappa says whether graders agree, not whether they are right: two graders can share a mistake. The readings (slight, fair, moderate …) are a convention. Weighted kappa for ordered labels is not implemented; ordinal scales are compared as plain categories. It is an AI reading a text file: it can be wrong, so check what matters against your own records and the source.
What’s in the buy.
CARD.md — the whole card: who it is, its laws, when it says no, what it carries · SKILL.md — the door your AI reads first · START-HERE.md — the first five minutes, for you
KAPPA-IN-PLAIN-WORDS.md — the guide in references/, in the card's own voice
scripts/agreement.py — the machine: kappa · fleiss · calibrate, standard library only, selftest 10/10
WITNESS.md + SELFTEST.md — every command on the record, as it printed · MANIFEST.md, every file hashed · AUDIT.md, the twelve checks
MY-LEDGER.md — the card's memory, on your own machine · TIN.md · LICENSE-NOTE.md · THE-CLAUSES.md · HOSTS.md
card.svg + card-machine.svg + card.json · card.html, the card as one offline file · the doors for ChatGPT, Gemini, Cursor, Claude Code and MCP hosts (rules/, system/, mcp/)
PROOF · SELFTEST 10/10 · 3 COMMANDS ON THE RECORD · WITNESSED IN CLAUDE 2026-10-09 · EDITION 1 · PUNCH 5A588E01324E4DA6AC4E34D0
ON EVERY DOOR · A CARD FOR YOUR AI, A FILE FOR YOUR COMPUTER
In Claude: turn on code execution under Settings › Capabilities, then Customize › Skills › + › Create skill, and upload the zip as it is — START-HERE.md in the box walks you through it, and tells you how to keep the ledger. In ChatGPT, Gemini and the rest: a Project, a GPT or a Gem with the card’s files attached and the short door in the box pasted as its instructions. The words run anywhere an AI reads Markdown; the machines run wherever the host runs Python. Witnessed in Claude only. The host marks →
Take it.
Nº 834 PROPOSED — LAUNCHING SOON
Sold as is under the term printed. A card instructs an AI you operate; results depend on your model, your files and your judgement. Not medical, legal, tax or financial advice, never the person’s boss, and not a licence to practise. The terms →
One payment. The file is yours to keep and keeps working at its edition; new editions issued in the year after you buy reach you by email as replacement files. Tax is added at checkout where it applies. Nothing auto-renews.
A TERM, NOT A SUBSCRIPTION: NOTHING RENEWS ITSELF AND NOBODY IS CHARGED AGAIN. WHEN A TERM ENDS THE FILES STAY ON YOUR COMPUTER AND STAY READABLE — FREEZE, NOT LOSS.
THE MACHINE · IN THE OPEN · FROM THE WITNESSED RUN OF 2026-10-09
agreement.py, every command showing its working.
A small Python program, standard library only. Your AI runs it; you can too. Every block below is pasted from the card’s own WITNESS.md, as printed.
kappa
two graders: agreement, Cohen's kappa, the table, every disagreement
fleiss
many graders: Fleiss' kappa and the items they split on
calibrate
the AI judge against people: agreement, kappa, leniency, the cases to read
selftest
10 checks that prove the machine; it must print 10/10
kappa · TWO GRADERS: AGREEMENT, COHEN'S KAPPA, THE TABLE, EVERY DISAGREEMENT
fleiss · MANY GRADERS: FLEISS' KAPPA AND THE ITEMS THEY SPLIT ON
$ python3 scripts/agreement.py fleiss three-graders.jsonl
AGREEMENT · 4 items · 3 graders each · labels A, B · mean pairwise agreement 67% · chance 50% · Fleiss' kappa 0.33 (fair)
label shares · A 50% · B 50%
NOTE · 4 items is a small set: the kappa is rough
SPLIT · 2 item(s) the graders did not all agree on: s2, s4
calibrate · THE AI JUDGE AGAINST PEOPLE: AGREEMENT, KAPPA, LENIENCY, THE CASES TO READ
$ python3 scripts/agreement.py calibrate judge.jsonl human.jsonl
CALIBRATION · AI judge against people · 50 cases · agreement 70% · kappa 0.00 (slight)
judge says pass 100% of the time · people say pass 70% · the judge is LENIENT by 30 points
false pass (judge yes, people no) · 15: case-36, case-37, case-38, case-39, case-40, case-41, case-42, case-43, case-44, case-45
missed pass (judge no, people yes) · 0: none
VERDICT · below substantial: read the disagreements, tighten the rubric, and have a person judge where it matters
THE NUMBERS, NAMES AND DATES IN THESE RUNS ARE THE STUDIO’S OWN EXAMPLES, INVENTED FOR THE RECORD; NO REAL COMPANY, PERSON OR PROPERTY IS NAMED.
QUESTIONS AT THE TABLE
Asked before the wire.
What does it need from me?
Two people grade the same 50 cases, blind. It asks first for the labels by case id from each grader (and from the AI judge), and the rubric they graded against.
Does it send anything anywhere?
No. The machine is a standard-library Python program that runs on your own computer and sends nothing anywhere; the words run wherever an AI reads Markdown. No server, no account, no key.
What is on the record?
The self-test (10/10) and every command run on real example inputs, pasted into WITNESS.md as they printed; the punch on the face is the SHA-256 of SKILL.md, so a downloaded copy can be checked against this room.
Can it be wrong?
Yes — it is an AI reading a text file. Kappa says whether graders agree, not whether they are right: two graders can share a mistake. The readings (slight, fair, moderate …) are a convention. Weighted kappa for ordered labels is not implemented; ordinal scales are compared as plain categories.
What will it never do?
It will never trust a rubric nobody has disagreed over; never read kappa without the raw agreement beside it; never let a lenient judge grade the launch.
