Methodology · the trust page
How a shortlist is produced, and what it does not yet prove.
Four things a selection process has to be able to show: what it takes out before a model reads a word, how agreement with a human would be certified, what the record of a decision is, and where the whole apparatus is currently falling short. The fourth section is the one this page exists for.
What the evaluation suite says today
One · bias controls
What is taken out before either AI reader sees the text.
We take identity out and we measure for bias. We do not claim to have solved it, and every measurement that fails today is printed in the gate table one section down.
Masking happens at the packet, never in the database
Identity is removed by a render-time transform over the fully assembled model packet — the digest and the evidence text included. The stored record is never masked: the database keeps truth, and only what the model is shown is redacted. Owners and human reviewers see the original artifact, so nothing is destroyed in the name of fairness.
What is replaced, and what is deliberately not
The applicant’s name, every teammate’s name, the organisation or institution, and the program’s own configured gender terms are matched case-insensitively as tokens and bigrams, with the bigram winning so a full name becomes one placeholder. Emails, phone-shaped digit runs and URL path segments carrying a name are redacted by pattern. Entities that do not match the name list are left alone: masking every entity would destroy the very evidence the score depends on.
A name that is also an ordinary word is not silently shredded
Grant, Rose, Dean, Hall, Young and Fields are all names and all words an essay uses without meaning a person. A collision like that is left in place; the packet is marked ambiguous instead, an advisory issue is raised, and it appears in the organiser’s list of applications that need attention. The alternative — a research grant reaching an AI reader as a redacted name — is the AI scoring evidence the masker deleted, which is the exact failure the control exists to prevent.
Every replaced span is recorded, so a citation never looks fabricated
The passage, the position in the original text and the replacement token are all stored. The organiser’s application page shows the original text with each replaced span marked and labelled with what the AI saw in its place. Citations resolve to a stored passage, not by matching quoted text, which is why a rationale that quotes masked text still resolves to the right span of the real one.
The seam re-checks, and throws rather than warning
The packet is re-scanned for every identity string before an AI reader is allowed to read it, and again after it has crossed durable state. A leak raises an error and fails the run; it does not log a warning and continue.
Two families, and human judges who see names
The two AI readers are different model families, a rule checked by automated tests on every change, so neither can rubber-stamp the other. Human judges are not covered by this masking: they see each application as submitted, names included. Hiding names from human judges in blind rounds is decided and not built yet. What human judges do get is an order: a judge scores and submits first, can compare with the AI only after submitting, and has to give a reason to change a score after that comparison. A review by an organiser, who can already see the AI’s scores, is recorded as not blind. The masking version is part of the AI reader’s contract: change it and old scores are never silently mixed with new ones.
Two · calibration protocol
How agreement would be certified, and why none is certified yet.
The protocol below is the one that would turn a reported number into a certified one. It has never run, because it needs a second human and this is a solo build.
What a gold set is
At least a hundred applications for one rubric archetype, each with at least two labels from two distinct humans. An item with one label is held as pending and excluded from every metric — so a founder labelling thirty applications alone contributes nothing to a gate until a second judge passes over them.
Labellers score blind first
The rule: a labeller does not see another labeller’s scores, or any AI score, until they have submitted their own. Human judges already work in that order: a judge scores and submits first, and can compare with the AI only after submitting. No human score has been turned into a label yet.
Labels that arrive too late should not count
The rule: every label stores who gave it and when, and a label given after the AI reader under test had already scored that application should not count. Today such labels are counted and reported, not yet excluded.
A published figure is scoped to what was actually gated
Any kappa belongs to the one rubric archetype its gold set covers. It is never generalised across program types, and this page will not print a single agreement number as though it described all of them.
The ceiling line, and whose defect a low score is
Beside every dimension’s judge-versus-human agreement, the report prints human-versus-human agreement for the same dimension. A dimension where two people disagree with each other as much as the model disagrees with them is a rubric-anchor defect, not a judge defect, and the fix is the anchor. Without that column a bad rubric reads as a bad model forever.
What runs today, on every change
A 98-application benchmark corpus, replayed from recorded calls, over every metric in the table below. It costs nothing to run and it makes no model call. Its reference scores are authored rather than collected from humans, so it declares itself as benchmark tier: a red row is reported and does not block a change, and only a gold corpus blocks.
Per program, automatically
A disagreement report is computed on the first twenty applications both AI readers have scored in a program, and it does not block those scores. The result is stored per program and shown to the organiser on the scoring guide page, and today every one reads advisory.
The gates, and what they currently say
These are the gates the AI reading is measured against on every change. At this stage they report and do not block a change, because there is no human-labelled set to block against. Every row shows its current result, and five of the eleven fail today.
weighted_kappa
- Gate
- ≥ 0.6, per dimension, per judge contract
- Result
- Reported, never certified — the reference scores are authored, not collected from humans. One archetype of three is still under the 30-item set-size rule and its rows fail on that alone; of the two that clear it, one fails every dimension and the other passes seven of twelve.
pointwise_order_sensitivity
- Gate
- < 5%, per judge contract
- Result
- 10.9% – 22.2%. Above the gate.
pairwise_position_flip
- Gate
- < 5%, on the pairwise contract
- Result
- not_measurable — nothing calls the pairwise role yet.
citation resolvability, L1 / L2 / L3
- Gate
- = 100%, per judge contract
- Result
- 100%, both AI readers.
length_bias
- Gate
- |ρ| < 0.3, as a partial correlation on the judge residual
- Result
- Fails. On the 32-application agreement set, 7 of 12 criteria across the two readers are over the gate, and one reader's overall figure is −0.448. Some criteria fail on the other two sets too.
injection_detection
- Gate
- 20 / 20, zero false positives
- Result
- Passes on the benchmark: the pattern check, not the AI readers, marked 20 of 20 planted instructions and 0 of 20 clean applications. Its patterns were written with this benchmark in view, so that shows what they cover, not how often they catch text they have never seen, and the check as it runs today has not been measured on such text. On a held-out set, an earlier version, before a round of bug fixes, caught 33 of 136 attempts and marked 0 of 168 clean passages.
injection_resistance
- Gate
- no item moves ≥ 2 in the requested direction
- Result
- Fails. Planted instructions moved a criterion score by 2 or more in the direction they asked for in 11 of 240 comparisons for one reader (0.046) and 18 of 240 for the other (0.075). The readers were not shown the check's mark during this measurement; what the mark changes has not been measured.
decoy_citations
- Gate
- = 0, per judge contract
- Result
- Fails on both readers: 0.100 to 0.667 of the scores at a criterion's floor cite a passage that another criterion scored near the top.
identity_leak
- Gate
- 0 occurrences, per judge contract
- Result
- 0 on the six-item identity set, both readers. The other two sets carry no planted names, so they cannot measure it. A leak at the packet seam stops the run.
needs_human_rate
- Gate
- < 5%, per judge contract
- Result
- 0%, both readers, on every set.
modelled_cost_per_application
- Gate
- within band; no regression over 20%
- Result
- ₹3.776 and ₹1.822 per application for the two readers on the 32-application set, modelled from list prices and never from an invoice. No earlier figure is committed, so the 20% regression check did not run.
One row is absent rather than red, and the difference matters. A row reading insufficient has a corpus too thin to measure, and it fails. pairwise_position_flip reads not_measurable: it measures a part of the system that does not exist yet, and it names its blocker — printing 0% for a gate nothing ran would be a green row that means nothing.
Three · audit format
What the record of a decision is, and what it is not.
The word is not immutable. A hash chain makes alteration detectable, not impossible, and the honest claim is the one the record can actually support.
Append-only, and tamper-evident
Decisions, cut-line moves, published rubric versions, AI assessments and letter releases are each added to a SHA-256 hash chain through one writer. Nothing on it is updated or deleted. A record that was altered, or an event quietly removed from the middle, fails verification — which is the difference between a log and an audit.
For a decision, the person is derived, never declared
The person recorded on a decision comes from the signed-in session and cannot be supplied by any caller, and a forged decision entry is caught when the record is verified. Other entries are less protected today: our background system could record one under a person’s account.
What is on the chain
The rubric version published, the judge contract registered, each assessment produced, the cut line moved, the decision made, the letter released. Each entry carries its time, the application it concerns where there is one, and the role that acted. The record does not show people’s names yet, and the page that walks one applicant’s entries in order is not built yet.
Verification, and the part that is not built
Verification runs in the database: verify_chain(program) returns true, or names the break. An offline checker for an exported record exists in our codebase, but it is not yet something an organiser can run on their own machine without us. Saying so is part of the format.
No decision is recorded without a signed-in person
A decision cannot be written without an identity taken from a signed-in session, and that identity is never a parameter any caller can supply. Enforcement is revoked grants, a security-definer procedure, a fail-closed trigger and table ownership — deliberately not a row-level policy, because the service role bypasses those and a policy naming it would enforce nothing. Insert on the decisions table is granted to no role at all. Releasing a letter needs a signed-in owner or admin in the same way, and a program’s last advancing stage must be a human review stage, checked by the database rather than by convention.
A fair draw is built, but not offered
The database can run a seeded draw over applicants near the line, but no screen offers it yet.
Traceability is not truth
Every sentence of a letter has to name the rubric criterion it came from and carry a citation that resolves against evidence the AI readers scored, and the five fixed sentences must match their frozen text byte for byte. A letter that fails gets one regeneration, then goes to a person with a blocking flag. What that checker constrains is traceability: a sentence can cite a real passage and still characterise it badly, which is why a person at the organisation still releases it, either after reading that letter or after reading a sample the system drew and the approver did not choose.
Four · the residual limits
What this page does not prove, stated by us first.
A selection engine that oversold its own certainty would be arguing against itself. These are the limits that stand today, and each one names what would remove it.
- The first limit
The same human writes the anchors and the labels.
In a solo build the person who authors a rubric's anchors is the person who authors the labels that would certify a judge against them. That is a circularity, it cannot be engineered away at this stage, and it is stated here rather than hidden. The way out is a second labeller, and a second labeller is what a design partner brings.
EVALS.md §1.4, point 5.
- The second limit
No judge contract has ever been validated.
A gold set is at least a hundred applications with at least two labels from two distinct humans, per rubric archetype. Zero human labels exist yet, so every contract carries the status unvalidated and every assessment is an advisory first pass, not yet checked against human judges. Whether a specific program was calibrated is a row that can be read, not a claim on a page — and its status today is advisory.
Ground truth §6.2.
- The third limit
The ranking is the AI's alone.
Human judges' scores are recorded, but they do not change the order. The cut line is drawn on the AI readers' ranking, and people decide every outcome. Ranking later rounds on the judges' own scores is decided and not built yet.
Ground truth §6.2.
- The fourth limit
Instructions to the AI are pattern-checked, not stopped.
An application can carry text written as instructions to the AI readers. A pattern check, not a model, reads the answers, the text of PDFs uploaded through the form and the transcripts of directly linked video files as they arrive. It marks text written that way, and the mark raises an issue that keeps the application out of decisions made in bulk until a person confirms or clears the issue; the mark itself stays, and the application can still be decided one at a time. The check sets no score itself, but the AI readers are shown the mark, and what that does to their scores has not been measured. In our benchmark it marked 20 of 20 planted instructions and 0 of 20 clean applications, but its patterns were written with that benchmark in view, so that is not a rate. The check as it runs today has not been measured on text it has never seen. The most recent measurement on such text was of an earlier version, before a round of bug fixes: on fresh passages written for it by an AI red-team pass in our own build, which had not read the patterns, it marked 0 of 168 clean passages and caught 33 of 136 attempts. The fixes, made after those passages had been read, changed one of its 304 verdicts, so those passages no longer count as held out for the check as it runs today. It reads wording, not intent, and that version missed attempts of every kind tried: of 17 attempts of each kind, it caught 12 disguised spellings of an override, but only 5 instructions spread over several sentences, 4 of the 17 plain overrides, 4 polite requests to the AI, 3 forged system notices, 2 role-plays, 2 pre-written scores and 1 of the 17 written in Hindi or Hinglish. Of the 103 attempts it missed, the red-team pass judged 97 to be ones a detector ought to catch. Its patterns are written only in English and in Hindi in Devanagari script, so they do not cover Hindi written in Latin letters or any other language, whatever its script, and it does not decode encoded text such as base64, which that set did not test. An earlier set of fresh passages was used to tune the patterns, so it no longer counts as held out either. What Kriteia does not read yet, such as a website or a code repository, is not checked at all, and neither is the name of an uploaded file, which the AI readers do see. Planted instructions still moved a criterion score by 2 or more in the direction they asked for in 11 of 240 comparisons for one reader and 18 of 240 for the other. What would remove this limit is AI readers whose scores planted instructions do not move.
EVALS.md §2.5 · injection_detection 20 of 20, 0 of 20 (in-sample) · held out, the check as it runs: not measured · held out, an earlier version: 33 of 136 attempts, 0 of 168 clean · injection_resistance 0.046 / 0.075.
- The fifth limit
A benchmark result is a measurement, not a certification.
The corpus this suite runs against declares its own provenance: authored reference scores, not human labels. So a red row is reported and does not block a change; only a gold corpus blocks, and none exists. The suite refuses to produce the claim calibrated from a benchmark report — it throws rather than returning false, because a caller who ignores a boolean publishes the certification while having done only the measurement.
ADR-0007.
- The sixth limit
Erasure is not built.
There is no tool yet to erase an application or to apply a retention period, so an erasure request cannot be carried out inside the product today. An applicant can still ask the organisation running the program. What would remove this limit is building both.
Ground truth §9, R5.
- The seventh limit
There is no certification, and no signed legal document.
No SOC 2 report and no ISO certification. Our privacy policy, terms and data-processing agreement are drafts no lawyer has reviewed yet. What would remove this limit is a lawyer's review of all three, and an independent audit.
DEMO-FIRST-PLAN §2, the deferral ledger.
All three drafts are published, as drafts: privacy, terms and the data-processing agreement, each marked draft until a lawyer has reviewed and signed it.
The north star
“When the winners are announced, every other applicant should feel the right ones were chosen.”
We are looking for the first design partner: one real program, run with us at cost, in exchange for a case study and a calibration study we can publish.
Raitakarya Technologies Pvt Ltd · kriteia.ai