What the checks see, and what they cannot
1253 exam-eligible items, 0 of them read by a person. 93 sequencing edges, 0 confirmed by a person. 0 of 901 knowledge points adopted by a person.
The checks, and their edges
Six checks run over the bank. Each is listed with what it can see and the specific thing it has been measured to miss, because a check whose limits are unstated reads as stronger than it is.
Fabrication gate
Sees. Whether the passage an item quotes is a character-for-character substring of the section it names.
Cannot see. Catch a real passage quoted from the wrong section for the question asked, or a passage that is genuine but does not support the claim.
Numeric check
Sees. Whether every figure in an explanation appears in the cited passage, and flags a conversion that replaces the source’s own unit.
Cannot see. Read negation. “The figure is 24 hours, not 24 days” was flagged for asserting 24 days. The explanation was reworded rather than teaching the check about negation, because negation handling was already removed once for being unreliable.
Modality rubric
Sees. Whether an explanation asserts an obligation where the cited passage grants a permission.
Cannot see. Tell which clause a modal belongs to. Six of seven flags it raised on the real bank were false positives: a partial quote omitting “are required to”, and one passage holding both “no person shall operate” and “the Minister may issue a NOTAM”. Each override is recorded on the item with a written reason and copied into the review log.
Contradiction scan
Sees. Two true/false items on the same rule with near-identical stems and opposite answers.
Cannot see. See anything wider. Two rules were tried and removed: “same rule, similar stem, different answer” gave 228 pairs that were entirely false positives, because a 3 NM airport rule and a 1 NM heliport rule are both correct and read alike.
Bank audit
Sees. Whole-bank properties no per-item check can see: answer-position bias, test-taking cues, duplicate items across banks, options repeated within an item.
Cannot see. Judge whether a distractor is plausible rather than merely wrong, which is the difference between a question and a trick.
Admission gate
Sees. Whether a structural edge’s evidence is verbatim against the section it cites, and whether a sequencing edge carries a confirmation naming a person.
Cannot see. Confirm anything itself. It refused all 21 sequencing edges until their provenance was corrected: every one claimed human authorship from a string hardcoded in the generator.
The corpus and the model
| corpus sections | 906 |
| concepts | 537 |
| knowledge points | 901 |
| graph edges | 951 |
| sequencing edges | 93 — 0 confirmed by a person |
| question facets in the bank | 8 |
How to check any of this yourself
Every claim on this site resolves to a section you can open. The search returns the corpus as it is stored, and a lesson’s source panel shows the passage each citation names.
Labels used here mean specific things. ai-reviewed means the same model family that wrote an item checked the claim against the passage it cites. reviewed means a person read it, and no item currently carries that label. derived means a document produced a record; agent and unattributed mean no person is named, and those are kept distinct from human because the gates check the label and a label a program can write does not admit anything.