Design the extraction confirmation step on both sides #16

New Issue

christian · 2026-08-02T03:05:54Z

christian commented

2026-08-02 03:05:54 +00:00

Question

Both the RFP and the response are extracted by a model. An extraction error in a rubric decides who wins a contract, which makes confirmation load-bearing rather than polish.

Resolve: what a retailer is shown to confirm and how corrections feed back; what a vendor is shown; how confidence is surfaced; what happens when someone confirms without reading; whether an unconfirmed RFP can go live at all; and how a correction after bids arrive is handled.

Added by a skeptical review pass

This ticket now owns the extraction completeness hole, recorded in
the schema decision: a requirement missed entirely by extraction never appears for anyone
to confirm. Framing worked out since, to resolve rather than inherit:

It is buyer-side. Extraction runs once, on the retailer's document, at issue. A miss means the
retailer awarded against an incomplete evaluation and the protest exposure is theirs.
Do not sell it as a vendor feature. A paid pre-submission completeness check would monetize
our own defect and would not reduce the retailer's exposure anyway. The legitimate adjacent paid
feature is a different thing: does your response fully address the RFP — response
completeness, not extraction completeness.
Span coverage is the primary mechanism, and the schema already carries it. Every requirement
has a provenance span, so regions of the source document that no requirement points at can be
computed deterministically. That converts an unknown unknown into a reviewable list.
Caveat: RFPs are full of non-requirement text, so uncovered regions are noisy. Tightening needs a
second, independent pass asking a different question (is there an obligation in this span),
not a re-run of the same extractor — two independent passes disagreeing is the actual signal.
Vendor reports are a secondary, biased detector. Vendors read the binding PDF and are the
cheapest place to notice a missing section, but they report omissions that help them and stay
quiet on ones that hurt them. Free, and recorded — a vendor who flagged an omission and was
ignored is a witness at protest.

Also added: criterion_requirement is authored, not extracted — the criterion-to-requirement
mapping is implicit in almost every real RFP, so this surface asks the retailer to create structure
the binding document does not contain. It needs its own attribution, and it is the one part of the
confirmation flow that is authoring rather than confirming. See the note on the schema ticket.

Parent: #1

## Question Both the RFP and the response are extracted by a model. An extraction error in a rubric decides who wins a contract, which makes confirmation load-bearing rather than polish. Resolve: what a retailer is shown to confirm and how corrections feed back; what a vendor is shown; how confidence is surfaced; what happens when someone confirms without reading; whether an unconfirmed RFP can go live at all; and how a correction after bids arrive is handled. ## Added by a skeptical review pass **This ticket now owns the extraction completeness hole**, recorded in [the schema decision](https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/10): a requirement missed entirely by extraction never appears for anyone to confirm. Framing worked out since, to resolve rather than inherit: - **It is buyer-side.** Extraction runs once, on the retailer's document, at issue. A miss means the retailer awarded against an incomplete evaluation and the protest exposure is theirs. - **Do not sell it as a vendor feature.** A paid pre-submission completeness check would monetize our own defect and would not reduce the retailer's exposure anyway. The legitimate adjacent paid feature is a different thing: *does your response fully address the RFP* — **response** completeness, not extraction completeness. - **Span coverage is the primary mechanism, and the schema already carries it.** Every requirement has a provenance span, so regions of the source document that no requirement points at can be computed deterministically. That converts an unknown unknown into a reviewable list. Caveat: RFPs are full of non-requirement text, so uncovered regions are noisy. Tightening needs a **second, independent** pass asking a different question (*is there an obligation in this span*), not a re-run of the same extractor — two independent passes disagreeing is the actual signal. - **Vendor reports are a secondary, biased detector.** Vendors read the binding PDF and are the cheapest place to notice a missing section, but they report omissions that help them and stay quiet on ones that hurt them. Free, and recorded — a vendor who flagged an omission and was ignored is a witness at protest. **Also added:** `criterion_requirement` is **authored, not extracted** — the criterion-to-requirement mapping is implicit in almost every real RFP, so this surface asks the retailer to create structure the binding document does not contain. It needs its own attribution, and it is the one part of the confirmation flow that is authoring rather than confirming. See the note on the schema ticket. --- Parent: #1

christian added the

wayfinder:grilling

wayfinder:ticket

labels 2026-08-02 03:05:54 +00:00

christian added a new dependency 2026-08-02 03:06:25 +00:00

#10 Define the common schema for RFPs and responses

christian referenced this issue

2026-08-02 15:22:59 +00:00

Define the common schema for RFPs and responses #10

christian referenced this issue

2026-08-02 15:25:06 +00:00

Skeptical review pass over all closed decisions #33

christian referenced this issue

2026-08-03 16:47:06 +00:00

Design the app framework — retailer and vendor surfaces #23

christian referenced this issue

2026-08-03 20:19:10 +00:00

Design AI bid ranking, scoring, and its defensibility #15

christian self-assigned this 2026-08-03 20:21:16 +00:00

christian commented

2026-08-03 20:44:04 +00:00

Resolution

Four decisions. The through-line: individual attention is spent exactly where a mistake can change
who wins, and nowhere else. Volume is what produces rubber-stamping, so every rule here is a rule
about what not to put in front of a human.

1. Issue is gated on outcome-changing requirements only

must be confirmed individually     mandatory gates
                                   criterion-mapped requirements
may be bulk-accepted               informational

Nothing issues until the first two groups are confirmed one at a time. On a 47-requirement RFP that
is roughly 39 real decisions rather than 47 or 8.

Why this line. It is the same taxonomy #15 routes on, so confirmation and evaluation agree
about what each requirement is. An informational requirement cannot disqualify anyone and cannot
move a score, so reviewing it individually buys nothing and costs attention that the gates deserve.

Why not confirm everything. 47 to 200 confirmations per event, most on text that cannot affect
anyone, is precisely how a confirmation step becomes a click-through — the burden defeats the
purpose. And this cost lands on the free side, which is the side we need to adopt.

Why not advisory. An unconfirmed rubric would decide an award, and #15 freezes the rubric map
at issue — an unconfirmed map would freeze wrong.

Ordering matters as much as scope. The ledger is ordered by extraction uncertainty, not
document order, so the least reliable items are seen first and while attention is freshest.
Confidence orders the queue; it never gates and never auto-accepts.

Rubber-stamping is not preventable, only attributable. Nothing here stops someone clicking
through 39 items. What the design does is keep the number small, put the worst first, offer bulk
acceptance only where nothing is at stake, and record a confirmer per requirement.

2. The completeness sweep gates on disagreement, not on coverage

Two signals, two treatments:

Blocking — a span the second pass says contains an obligation, which no requirement covers.
Resolve by adding a requirement or recording not an obligation, which is a named human call.
Report — uncovered spans with no obligation asserted. Listed, never gating.

Why not a coverage percentage. RFPs are mostly non-requirement prose, so a coverage threshold
measures document style rather than extraction quality: a 55-page RFP with a 20-page appendix never
clears it and a terse one always does. Worse, it rewards padding the rubric with requirements that
exist only to cover text.

The second pass must be genuinely independent — a different question (is there an obligation in
this span) rather than a re-run of the extractor. Two independent passes disagreeing is the entire
signal; sharing a prompt or a code path destroys it. This is a constraint on #35, not a style
preference.

This also closes the miskinding hole decision 1 opens. The sweep runs over spans covered only
by informational requirements as well as uncovered ones, so an obligation extracted but miskinded as
informational — which would otherwise be bulk-accepted and never seen — surfaces as a blocking
disagreement.

Recorded as unfixed: a miss both passes share is still invisible. Independence reduces
correlated failure; it does not eliminate it.

Vendor omission reports are the free secondary detector, and biased as #33 framed: vendors
report omissions that help them and stay quiet on ones that hurt them. A report becomes a ledger
item on the retailer's side, classified by decision 3 like any other correction. Recording it is the
point — a vendor who flagged an omission and was ignored is a witness at protest.

3. Corrections classify on whether a bidder would answer differently

That is the test, not whether the text changed.

Substantive — an obligation added, removed or materially changed. Extends the deadline,
notifies every participant, and reopens only the affected answer on already-submitted
responses. Everything else in those responses is untouched.
Clarification — wording, a transcription fix, a typo in an extracted span. Recorded,
participants notified, no deadline effect.

This makes #23's reissue-or-clarify fork concrete: the fork exists, and this is the rule that
picks a branch.

Why not immutable-and-reissue. Real procurement issues addenda constantly; restarting every
bidder's clock over a typo is not how the domain works, and the pressure it creates is to correct
off-platform by email — which loses the record the venue exists to hold.

Why not log-and-change-nothing. A vendor who submitted on the 8th would be judged against a
requirement that did not exist when they answered. #12 already declined that pattern's cousin,
mid-RFP audience widening, for the same fairness reason.

A property worth naming, because it falls out of #14 rather than being designed here: sealed
bidding makes mid-flight amendment safe. The retailer has read nothing, so an amendment cannot be
tailored to advantage a bidder whose response they have already seen. Under any
readable-as-it-arrives model this decision would be dangerous.

4. The vendor resolves indeterminates and acknowledges gate failures; the rest is bulk-attested

Symmetric with decision 1 — individual attention only where the outcome moves.

Resolve individually — every indeterminate answer. #10 puts the attestation of one's own
indeterminates on the vendor, and this is where that happens.
Acknowledge explicitly — every unanswered mandatory requirement, with the consequence
stated: leaving this blank fails a pass/fail gate.
Bulk-attest — the answered remainder, in one act.

Where the paid line falls, stated precisely. The gate warning and the coverage state are
deterministic facts we have already computed, so they are free — we never charge a vendor to
learn they are about to be disqualified. The Pro feature next door is judgement: is this answer
actually sufficient for this requirement. That is response completeness, which #33
distinguished from extraction completeness for good reason — the latter is our own defect and
monetizing it would be indefensible.

Why not one global attestation. It is the strongest single legal artifact and it wastes the one
moment where a cheap fix is still possible; indeterminates would reach the retailer unresolved.

Why not attest all 47. Same rubber-stamping failure as the retailer side, aimed at the side that
is paying us.

Consequences that did not need asking

Requirement kind is now a confirmed field, not a label. #15 routes decisions on it, so
miskinding is a correctness bug. Kind is displayed and confirmed alongside the requirement text.

Confirmation captures the document's own scale. Points, labels, and the prose anchor per point,
per #22 — the anchor is what the model matches against later, and it cannot be recovered after
issue.

Authoring is labelled as authoring. The criterion-to-requirement mapping sits in this same flow
but is not confirmation — the retailer is creating structure the binding document does not contain.
#15 froze it at issue and gave it attribution; this surface must not let it blend visually into
the rows that have spans behind them.

Constraints handed to other tickets

#21 audit — confirmation events with a confirmer per requirement, not an obligation
dismissals with their author, amendment classification and the reopened-answer set, and vendor
omission reports with whether they were acted on. The last is the strongest protest artifact on
the map.
#35 layout — the completeness sweep is a separate module from the extractor, sharing no
prompt and no code path, or it stops being a second opinion.
#30 corpus — must include a solicitation with an obligation buried where a naive extractor
misses it, or the sweep ships unexercised. Labelled known-misses are the only way to measure its
recall at all.
#23 surfaces (closed) — the confirmation ledger is a compose-surface object ordered by
uncertainty, and the vendor's pre-submit resolution screen is a distinct state of the bid surface.
#18 integration — an amendment is an outbound event, not only an in-app state change.

## Resolution Four decisions. The through-line: **individual attention is spent exactly where a mistake can change who wins, and nowhere else.** Volume is what produces rubber-stamping, so every rule here is a rule about what *not* to put in front of a human. ### 1. Issue is gated on outcome-changing requirements only ``` must be confirmed individually mandatory gates criterion-mapped requirements may be bulk-accepted informational ``` Nothing issues until the first two groups are confirmed one at a time. On a 47-requirement RFP that is roughly 39 real decisions rather than 47 or 8. **Why this line.** It is the same taxonomy https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15 routes on, so confirmation and evaluation agree about what each requirement *is*. An informational requirement cannot disqualify anyone and cannot move a score, so reviewing it individually buys nothing and costs attention that the gates deserve. **Why not confirm everything.** 47 to 200 confirmations per event, most on text that cannot affect anyone, is precisely how a confirmation step becomes a click-through — the burden defeats the purpose. And this cost lands on the free side, which is the side we need to adopt. **Why not advisory.** An unconfirmed rubric would decide an award, and https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15 freezes the rubric map at issue — an unconfirmed map would freeze wrong. **Ordering matters as much as scope.** The ledger is ordered by **extraction uncertainty, not document order**, so the least reliable items are seen first and while attention is freshest. Confidence orders the queue; it never gates and never auto-accepts. **Rubber-stamping is not preventable, only attributable.** Nothing here stops someone clicking through 39 items. What the design does is keep the number small, put the worst first, offer bulk acceptance *only* where nothing is at stake, and record a confirmer per requirement. ### 2. The completeness sweep gates on disagreement, not on coverage Two signals, two treatments: - **Blocking** — a span the second pass says contains an obligation, which no requirement covers. Resolve by adding a requirement or recording **not an obligation**, which is a named human call. - **Report** — uncovered spans with no obligation asserted. Listed, never gating. **Why not a coverage percentage.** RFPs are mostly non-requirement prose, so a coverage threshold measures document style rather than extraction quality: a 55-page RFP with a 20-page appendix never clears it and a terse one always does. Worse, it rewards padding the rubric with requirements that exist only to cover text. **The second pass must be genuinely independent** — a different question (*is there an obligation in this span*) rather than a re-run of the extractor. Two independent passes disagreeing is the entire signal; sharing a prompt or a code path destroys it. This is a constraint on https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/35, not a style preference. **This also closes the miskinding hole** decision 1 opens. The sweep runs over spans covered *only* by informational requirements as well as uncovered ones, so an obligation extracted but miskinded as informational — which would otherwise be bulk-accepted and never seen — surfaces as a blocking disagreement. **Recorded as unfixed:** a miss both passes share is still invisible. Independence reduces correlated failure; it does not eliminate it. **Vendor omission reports are the free secondary detector**, and biased as https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/33 framed: vendors report omissions that help them and stay quiet on ones that hurt them. A report becomes a ledger item on the retailer's side, classified by decision 3 like any other correction. Recording it is the point — **a vendor who flagged an omission and was ignored is a witness at protest.** ### 3. Corrections classify on whether a bidder would answer differently That is the test, not whether the text changed. - **Substantive** — an obligation added, removed or materially changed. Extends the deadline, notifies every participant, and **reopens only the affected answer** on already-submitted responses. Everything else in those responses is untouched. - **Clarification** — wording, a transcription fix, a typo in an extracted span. Recorded, participants notified, no deadline effect. This makes https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/23's *reissue-or-clarify* fork concrete: the fork exists, and this is the rule that picks a branch. **Why not immutable-and-reissue.** Real procurement issues addenda constantly; restarting every bidder's clock over a typo is not how the domain works, and the pressure it creates is to correct off-platform by email — which loses the record the venue exists to hold. **Why not log-and-change-nothing.** A vendor who submitted on the 8th would be judged against a requirement that did not exist when they answered. https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/12 already declined that pattern's cousin, mid-RFP audience widening, for the same fairness reason. **A property worth naming, because it falls out of https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/14 rather than being designed here:** sealed bidding makes mid-flight amendment safe. The retailer has read nothing, so an amendment cannot be tailored to advantage a bidder whose response they have already seen. Under any readable-as-it-arrives model this decision would be dangerous. ### 4. The vendor resolves indeterminates and acknowledges gate failures; the rest is bulk-attested Symmetric with decision 1 — individual attention only where the outcome moves. - **Resolve individually** — every `indeterminate` answer. https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/10 puts the attestation of one's own indeterminates on the vendor, and this is where that happens. - **Acknowledge explicitly** — every unanswered **mandatory** requirement, with the consequence stated: *leaving this blank fails a pass/fail gate.* - **Bulk-attest** — the answered remainder, in one act. **Where the paid line falls, stated precisely.** The gate warning and the coverage state are **deterministic facts we have already computed**, so they are free — we never charge a vendor to learn they are about to be disqualified. The Pro feature next door is judgement: *is this answer actually sufficient for this requirement*. That is **response** completeness, which https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/33 distinguished from extraction completeness for good reason — the latter is our own defect and monetizing it would be indefensible. **Why not one global attestation.** It is the strongest single legal artifact and it wastes the one moment where a cheap fix is still possible; indeterminates would reach the retailer unresolved. **Why not attest all 47.** Same rubber-stamping failure as the retailer side, aimed at the side that is paying us. ### Consequences that did not need asking **Requirement kind is now a confirmed field, not a label.** https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15 routes decisions on it, so miskinding is a correctness bug. Kind is displayed and confirmed alongside the requirement text. **Confirmation captures the document's own scale.** Points, labels, and the prose anchor per point, per https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/22 — the anchor is what the model matches against later, and it cannot be recovered after issue. **Authoring is labelled as authoring.** The criterion-to-requirement mapping sits in this same flow but is not confirmation — the retailer is creating structure the binding document does not contain. https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15 froze it at issue and gave it attribution; this surface must not let it blend visually into the rows that have spans behind them. ### Constraints handed to other tickets - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/21 audit** — confirmation events with a confirmer per requirement, *not an obligation* dismissals with their author, amendment classification and the reopened-answer set, and vendor omission reports with whether they were acted on. The last is the strongest protest artifact on the map. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/35 layout** — the completeness sweep is a **separate module from the extractor**, sharing no prompt and no code path, or it stops being a second opinion. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/30 corpus** — must include a solicitation with an obligation buried where a naive extractor misses it, or the sweep ships unexercised. Labelled known-misses are the only way to measure its recall at all. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/23 surfaces** (closed) — the confirmation ledger is a compose-surface object ordered by uncertainty, and the vendor's pre-submit resolution screen is a distinct state of the bid surface. - **https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/18 integration** — an amendment is an outbound event, not only an in-app state change.

christian closed this issue

2026-08-03 20:44:04 +00:00

christian referenced this issue

2026-08-03 20:44:04 +00:00

Retail RFP transmission system — v1 build spec #1

christian referenced this issue

2026-08-03 21:49:21 +00:00

Build the synthesized response corpus #30

christian referenced this issue

2026-08-03 21:55:08 +00:00

Build the synthesized response corpus #30

christian referenced this issue