christian/helmdocs-proposal-system
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

Design AI bid ranking, scoring, and its defensibility #15

New Issue
Closed
opened 2026-08-02 03:05:54 +00:00 by christian · 2 comments
christian commented 2026-08-02 03:05:54 +00:00
Owner
Copy Link

Question

Bids are ranked against the retailer's rubric. Procurement awards get challenged, so a score that cannot be explained is a liability rather than a feature.

Resolve: how a rubric criterion becomes a score; where deterministic computation ends and model judgment begins; how a score is explained back to the evaluator with citations into the response; how a human overrides and how the override is recorded; how ties and missing responses are handled; and how ranking stays stable when the model changes underneath it.

Prior art: the go-no-go agent's scoring and the deterministic-first, LLM-fallback pattern already used in packages/ai/src/agents/go-no-go/.

Parent: #1

## Question Bids are ranked against the retailer's rubric. Procurement awards get challenged, so a score that cannot be explained is a liability rather than a feature. Resolve: how a rubric criterion becomes a score; where deterministic computation ends and model judgment begins; how a score is explained back to the evaluator with citations into the response; how a human overrides and how the override is recorded; how ties and missing responses are handled; and how ranking stays stable when the model changes underneath it. Prior art: the go-no-go agent's scoring and the deterministic-first, LLM-fallback pattern already used in `packages/ai/src/agents/go-no-go/`. --- Parent: #1
christian added the
wayfinder:grilling
wayfinder:ticket
 labels 2026-08-02 03:05:54 +00:00
christian added a new dependency 2026-08-02 03:06:25 +00:00
#10 Define the common schema for RFPs and responses
christian added a new dependency 2026-08-02 03:06:26 +00:00
#21 Design the audit trail and evidentiary record
christian self-assigned this 2026-08-03 19:39:50 +00:00
christian commented 2026-08-03 20:19:10 +00:00
Author
Owner
Copy Link

Resolution

Four decisions. The through-line: every number that could decide an award is either arithmetic or
a human act, and never a model output. The model does real work throughout — locating, citing,
proposing — but there is no path by which its judgement becomes the basis of an award without a
person committing it.

1. Suggestion and score are separate objects; only a commit creates a score

suggestion                          score  (of record)
  response_id, criterion_id           response_id, criterion_id
  proposed_points                     points
  anchor_matched   "Very Good"        committed_by      <person>
  citations[]      (doc, page, span)  committed_at
  confidence                          derived_from_suggestion?
  model_version

score rows are always human-attributed. A suggestion is evidence with a proposal attached; it
is never promoted automatically, and there is no configuration that promotes it.

anchor_matched, not just a number. #22 found that scales are per-document with incompatible
label sets — Friendship's 3 Average and Salt Lake County's 3 Acceptable are the same integer and
different judgements — and that the prose anchor is the load-bearing part. So the model matches
an anchor and the suggestion records which one; a bare integer would be uninterpretable across
documents and is exactly the normalisation #10 bans.

Why not AI-scores-with-override. It makes an unreviewed model output the default basis of an
award, which is the liability this ticket exists to remove, and "who scored this?" answers "nobody
looked." It also lets a model upgrade silently rewrite the basis of comparison between vendors.

Why not human-only scoring. It discards the retailer-side pitch entirely, and adds almost
nothing over the coverage matrix #23 already draws.

What it costs, stated plainly. An evaluator can rubber-stamp eleven criteria across forty bids
and call it review. Nothing here prevents that — it only ensures the record says a person did it,
and names them.

2. Routing is by requirement kind; the model is barred from arithmetic and gates

Over the six structurally distinct kinds #22 identified:

mandatory gate          code    present / signed  -> pass | fail
certification form      code    fixed external schema
compliance schedule     code    constraint arithmetic (MBE nesting)
pricing                 code    rate function x declared scenario
-----------------------------------------------------------------
scored criterion        model proposes -> human commits
narrative               model proposes -> human commits

The model may LOCATE in every lane — "the signed AD-1048 is on p.31" is useful and safe. It
may DECIDE in only the bottom two.

Why the hard bar rather than deterministic-first-with-fallback, despite that being the proven
pattern in packages/ai/src/agents/go-no-go/. Two cases make routing-by-kind strictly better here:

  • #22 found frequency and quantity multipliers hide in footnotes, not table columns, and that
    dropping one silently converts an annual cost into a unit price. A fallback path hands exactly
    that arithmetic to a model at exactly the moment it is ambiguous, and the error reads as
    high-confidence. Arithmetic is code, always.
  • A mandatory gate disqualifies regardless of score. An ambiguous gate must escalate to a
    human, not to judgement. A model false-negative there eliminates a vendor from an award.

Also: under fallback, which lane a decision took depends on runtime luck, which is not something you
can explain in a protest.

What it costs. A miskinded requirement routes wrong, which puts real weight on #16 — kind is
now a load-bearing confirmation field, not a label.

3. No ordering exists until scores are committed

Rank is computed only from committed scores, using the document's own weights and stored
fractionally — #22's real tabulation separates the winner from the runner-up by 71.81 to 70.89, so
rounding would erase the actual result.

Before commit the surface offers triage, not rank: gate status, mandatory coverage, and derived
cost. Every one of those columns is deterministic; no model judged any of them.

Why. A published provisional ordering anchors the evaluator who is about to score, and the
anchoring leaves no trace in the audit trail — "the AI ranked us fourth and then the humans agreed"
is close to unanswerable. It is the same reasoning that made two-envelope worth taking seriously in
#14, applied to judgement rather than price. The blended variant is worse still: half human, half
model, in a ratio that changes hourly.

Within-document totalling is not normalisation. #10 bans cross-RFP scores and global ratings;
summing committed points by the RFP's own published weights is the retailer's own arithmetic, and
every real rubric in the research is exactly that.

Disqualification is not a rank. Gates are evaluated before scoring, so a gate-failed response is
shown separately rather than ranked last — ranking it implies it was in contention.

What it costs. The "rank my bids" demo needs scoring done first.

4. The criterion -> requirement mapping is frozen at issue; amending it voids affected scores

criterion_requirement is the one object in the schema with no span to point at — #33 found it
is authored, not extracted. It is therefore published with the rubric and frozen at issue, the same
treatment #12 gives the audience and for the same reason.

Amendment stays possible and expensive: it invalidates every committed score under the old
mapping and records the amendment and its author.

Why. A mapping authored after the bids are readable is how an award gets rigged — choose which
requirements feed a criterion once you know who answered what. Mutable-and-logged does not fix this:
a log shows the mapping changed, not that the change was innocent. Freezing makes it impossible to
do quietly, and the amendment path means a genuine mistake does not force a reissue that restarts
every bidder's clock.

Consequence for explanations. A score explains itself in two registers, and they are visibly
different: document-derived evidence (citations with page and span) and human-authored
structure (which requirements the retailer attached to this criterion, by whom, frozen when).

Consequence for surfaces. Authoring the mapping belongs on the compose surface, before
issue — not on the evaluation surface. That is a real addition to #23.

Consequences that did not need asking

Stability across model change is already solved by decision 1. Suggestions carry
model_version; committed scores are never re-derived. Re-running produces new suggestions that
cannot alter a committed score. A mid-evaluation model upgrade leaves some criteria scored under one
version and some under another — visible, and harmless, because the model authored neither.

Ties get no automatic tiebreak. Fractional scoring makes exact ties rare; when one occurs the
retailer decides, and that decision is a recorded human act like any other commit. Inventing a
tiebreak rule would be us making a judgement on their behalf.

Absent evidence produces no suggestion. Where every requirement mapped to a criterion is
unanswered, the model proposes nothing and the criterion is flagged no evidence found — rather
than proposing the scale's minimum, which would be a judgement dressed as a default.

Bid cost is not contract value. #22 found requirements contracts carry estimate-only
quantities, so derived_cost is a comparison figure against the declared scenario and must be
labelled as such wherever it appears. Ranking that presents it as contract value misrepresents
comparability.

Ranking may surface nothing #14 does not release. No cross-RFP metric, no win rate, no
vendor-visible rank — including their own.

Constraints handed to other tickets

  • #16 extraction confirmation — requirement kind now routes decisions, so miskinding is a
    correctness bug rather than a labelling one. Confirmation must also capture the document's own
    scale: points, labels, and the prose anchor per point.
  • #21 audit — commit events, amendment events with the voided-score set, gate results, and
    model_version per suggestion are all protest surface. This ticket's freeze-and-void rule is the
    main thing an audit reconstructs.
  • #29 rate structures — derived_cost is deterministic, so the v1 rate structure types must be
    closed and computable. A rate shape the code cannot evaluate has no fallback, by decision 2.
  • #35 layout — the enumerated bar on what a model may decide belongs beside the privileged-path
    list. Both are short, reviewable, and load-bearing.
  • #23 surfaces (closed) — two additions: rubric mapping is authored on the compose surface, and
    the evaluation surface has two states, triage before commits and rank after.
## Resolution Four decisions. The through-line: **every number that could decide an award is either arithmetic or a human act, and never a model output.** The model does real work throughout — locating, citing, proposing — but there is no path by which its judgement becomes the basis of an award without a person committing it. ### 1. Suggestion and score are separate objects; only a commit creates a score ``` suggestion score (of record) response_id, criterion_id response_id, criterion_id proposed_points points anchor_matched "Very Good" committed_by <person> citations[] (doc, page, span) committed_at confidence derived_from_suggestion? model_version ``` `score` rows are **always human-attributed**. A suggestion is evidence with a proposal attached; it is never promoted automatically, and there is no configuration that promotes it. **`anchor_matched`, not just a number.** https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/22 found that scales are per-document with incompatible label sets — Friendship's `3 Average` and Salt Lake County's `3 Acceptable` are the same integer and different judgements — and that **the prose anchor is the load-bearing part**. So the model matches an anchor and the suggestion records which one; a bare integer would be uninterpretable across documents and is exactly the normalisation https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/10 bans. **Why not AI-scores-with-override.** It makes an unreviewed model output the default basis of an award, which is the liability this ticket exists to remove, and "who scored this?" answers "nobody looked." It also lets a model upgrade silently rewrite the basis of comparison between vendors. **Why not human-only scoring.** It discards the retailer-side pitch entirely, and adds almost nothing over the coverage matrix https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/23 already draws. **What it costs, stated plainly.** An evaluator *can* rubber-stamp eleven criteria across forty bids and call it review. Nothing here prevents that — it only ensures the record says a person did it, and names them. ### 2. Routing is by requirement kind; the model is barred from arithmetic and gates Over the six structurally distinct kinds https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/22 identified: ``` mandatory gate code present / signed -> pass | fail certification form code fixed external schema compliance schedule code constraint arithmetic (MBE nesting) pricing code rate function x declared scenario ----------------------------------------------------------------- scored criterion model proposes -> human commits narrative model proposes -> human commits ``` **The model may LOCATE in every lane** — "the signed AD-1048 is on p.31" is useful and safe. **It may DECIDE in only the bottom two.** **Why the hard bar rather than deterministic-first-with-fallback**, despite that being the proven pattern in `packages/ai/src/agents/go-no-go/`. Two cases make routing-by-kind strictly better here: - https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/22 found **frequency and quantity multipliers hide in footnotes**, not table columns, and that dropping one silently converts an annual cost into a unit price. A fallback path hands exactly that arithmetic to a model at exactly the moment it is ambiguous, and the error reads as high-confidence. Arithmetic is code, always. - A **mandatory gate disqualifies regardless of score**. An ambiguous gate must escalate to a *human*, not to judgement. A model false-negative there eliminates a vendor from an award. Also: under fallback, which lane a decision took depends on runtime luck, which is not something you can explain in a protest. **What it costs.** A miskinded requirement routes wrong, which puts real weight on https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/16 — kind is now a load-bearing confirmation field, not a label. ### 3. No ordering exists until scores are committed Rank is computed **only** from committed scores, using the document's own weights and stored fractionally — https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/22's real tabulation separates the winner from the runner-up by 71.81 to 70.89, so rounding would erase the actual result. Before commit the surface offers **triage, not rank**: gate status, mandatory coverage, and derived cost. Every one of those columns is deterministic; no model judged any of them. **Why.** A published provisional ordering anchors the evaluator who is about to score, and the anchoring leaves no trace in the audit trail — "the AI ranked us fourth and then the humans agreed" is close to unanswerable. It is the same reasoning that made two-envelope worth taking seriously in https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/14, applied to judgement rather than price. The blended variant is worse still: half human, half model, in a ratio that changes hourly. **Within-document totalling is not normalisation.** https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/10 bans cross-RFP scores and global ratings; summing committed points by the RFP's own published weights is the retailer's own arithmetic, and every real rubric in the research is exactly that. **Disqualification is not a rank.** Gates are evaluated before scoring, so a gate-failed response is shown separately rather than ranked last — ranking it implies it was in contention. **What it costs.** The "rank my bids" demo needs scoring done first. ### 4. The criterion -> requirement mapping is frozen at issue; amending it voids affected scores `criterion_requirement` is the one object in the schema with **no span to point at** — https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/33 found it is authored, not extracted. It is therefore published with the rubric and frozen at issue, the same treatment https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/12 gives the audience and for the same reason. Amendment stays possible and expensive: it **invalidates every committed score under the old mapping** and records the amendment and its author. **Why.** A mapping authored *after* the bids are readable is how an award gets rigged — choose which requirements feed a criterion once you know who answered what. Mutable-and-logged does not fix this: a log shows the mapping changed, not that the change was innocent. Freezing makes it impossible to do quietly, and the amendment path means a genuine mistake does not force a reissue that restarts every bidder's clock. **Consequence for explanations.** A score explains itself in two registers, and they are visibly different: **document-derived evidence** (citations with page and span) and **human-authored structure** (which requirements the retailer attached to this criterion, by whom, frozen when). **Consequence for surfaces.** Authoring the mapping belongs on the **compose** surface, before issue — not on the evaluation surface. That is a real addition to https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/23. ### Consequences that did not need asking **Stability across model change is already solved by decision 1.** Suggestions carry `model_version`; committed scores are never re-derived. Re-running produces new suggestions that cannot alter a committed score. A mid-evaluation model upgrade leaves some criteria scored under one version and some under another — visible, and harmless, because the model authored neither. **Ties get no automatic tiebreak.** Fractional scoring makes exact ties rare; when one occurs the retailer decides, and that decision is a recorded human act like any other commit. Inventing a tiebreak rule would be us making a judgement on their behalf. **Absent evidence produces no suggestion.** Where every requirement mapped to a criterion is unanswered, the model proposes nothing and the criterion is flagged *no evidence found* — rather than proposing the scale's minimum, which would be a judgement dressed as a default. **Bid cost is not contract value.** https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/22 found requirements contracts carry estimate-only quantities, so `derived_cost` is a comparison figure against the declared scenario and must be labelled as such wherever it appears. Ranking that presents it as contract value misrepresents comparability. **Ranking may surface nothing https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/14 does not release.** No cross-RFP metric, no win rate, no vendor-visible rank — including their own. ### Constraints handed to other tickets - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/16 extraction confirmation** — requirement *kind* now routes decisions, so miskinding is a correctness bug rather than a labelling one. Confirmation must also capture the document's own scale: points, labels, and the prose anchor per point. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/21 audit** — commit events, amendment events with the voided-score set, gate results, and `model_version` per suggestion are all protest surface. This ticket's freeze-and-void rule is the main thing an audit reconstructs. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/29 rate structures** — `derived_cost` is deterministic, so the v1 rate structure types must be closed and computable. A rate shape the code cannot evaluate has no fallback, by decision 2. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/35 layout** — the enumerated bar on what a model may decide belongs beside the privileged-path list. Both are short, reviewable, and load-bearing. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/23 surfaces** (closed) — two additions: rubric mapping is authored on the compose surface, and the evaluation surface has two states, triage before commits and rank after.
christian commented 2026-08-03 22:51:09 +00:00
Author
Owner
Copy Link

Amendment: what a score commits against

#10 now fixes the scoring level, which this ticket's commit model needed and did not have.

A score commits at any criterion with NO WEIGHTED CHILDREN - derived from the rubric, not
configured. For Friendship PCS that is 10 criteria of 12; "Method of Approach" and "Experience" are
parents and are not themselves scored.

This interacts with binding. Friendship's response format gives one prose block per top-level
section, so an evaluator scoring FOOD_PACKAGING, USDA_FOODS and GEO_PREF is reading the same
passage three times with different questions. That is what the document's own weights demand, and it
is a real argument for the per-criterion suggestion this ticket already specifies - three
suggestions over one span, each citing it, is exactly the shape.

Also relevant to gate routing: mandatory_gate is decided by code, and a gate that no response
document can establish would otherwise return a confident wrong answer. gateEvaluableFromResponse()
excludes those. See the amendment on #10.

## Amendment: what a score commits against https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/10 now fixes the scoring level, which this ticket's commit model needed and did not have. **A score commits at any criterion with NO WEIGHTED CHILDREN** - derived from the rubric, not configured. For Friendship PCS that is 10 criteria of 12; "Method of Approach" and "Experience" are parents and are not themselves scored. **This interacts with binding.** Friendship's response format gives **one prose block per top-level section**, so an evaluator scoring `FOOD_PACKAGING`, `USDA_FOODS` and `GEO_PREF` is reading the same passage three times with different questions. That is what the document's own weights demand, and it is a real argument for the per-criterion suggestion this ticket already specifies - three suggestions over one span, each citing it, is exactly the shape. **Also relevant to gate routing:** `mandatory_gate` is decided by code, and a gate that no response document can establish would otherwise return a confident wrong answer. `gateEvaluableFromResponse()` excludes those. See the amendment on https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/10.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
HITL: resolved by conversation
wayfinder:map
The shared map for a wayfinding effort
wayfinder:prototype
HITL: resolved by making something rough
wayfinder:research
AFK: resolved by a research subagent
wayfinder:task
Manual work that unblocks a decision
wayfinder:ticket
A wayfinder decision ticket
No Label
wayfinder:grilling
wayfinder:ticket
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
christian
No Assignees christian
1 Participants
Notifications
Due Date
No due date set.
Blocks
#21 Design the audit trail and evidentiary record
christian/helmdocs-proposal-system
Depends on
#10 Define the common schema for RFPs and responses
christian/helmdocs-proposal-system
Reference: christian/helmdocs-proposal-system#15
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?