Add three synthesized Friendship PCS responses with ground truth

Covers all nine stress cases from #25, #15 and #16. The lowest bid is
disqualified on two missing health documents and the highest bid is
strongest on the SFA's own geographic-preference criterion, so a ranking
that cannot represent absence gets the field exactly backwards.

Deliberate traps: a planted $400 arithmetic error in a stated total, an
unverifiable total whose unit price lives only in the external Excel, an
unlabelled exception, a non-priceable exception that is really a conditional
withdrawal, and an answered-but-negative response.

Second schema finding, surfaced by the validator: R-B6C is a mandatory gate
that no response document can establish. Coverage over it is unknowable
rather than absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Christian
2026-08-03 18:39:08 -04:00
parent e81da40947
commit 19e7ffe92a
9 changed files with 1509 additions and 71 deletions

View File

@@ -63,3 +63,62 @@ present, its content is not, and no amount of better PDF parsing recovers it.
149 × "shall", 64 × "must" across 52 pages. An earlier ad-hoc extraction reported
zero of each; that was the extractor failing, not the document.
---
# Response set
Three synthesized responses in `responses/source/`, each paired with a truth
file. Designed so the field has a real outcome rather than three
interchangeable bids.
| Vendor | Binding | Price | Gates | Distinguishing feature |
|---|---|---|---|---|
| Potomac Provisions | tabbed compliance matrix | $589,970 | PASS | Complete; volume-banded pricing; 18% local |
| Chesapeake Food Service | free prose, no anchors | $541,280 | **FAIL** | Cheapest and disqualified; unlabelled exception |
| Anacostia Farm Collective | named sections + enumeration | $624,315 | PASS | 61% local; three labelled exceptions; seasonal pricing |
The lowest bid is disqualified and the highest bid is strongest on the
criterion the SFA wrote into its own rubric. That is deliberate: a ranking
that cannot represent absence puts Chesapeake first.
## Deliberate traps
**Planted arithmetic error (Potomac).** The stated base-year total is
$589,970; band price times scenario volume is $589,570. A $400 gap. #15
decision 2 puts all price arithmetic in code precisely so a stated total is
never trusted over a computed one — this is the case that catches an
extractor lifting the vendor's own figure.
**Unverifiable total (Chesapeake).** The case price exists only in the
external Excel sheet, so no computed total can be produced at all. Contrast
Potomac, where the arithmetic can be checked and fails.
**Unlabelled exception (Chesapeake).** A price-relief request phrased as
politeness inside the Pricing section, with no heading and no delta.
Anacostia's three exceptions are explicitly headed with deltas and act as
the control.
**Non-priceable exception (Anacostia, EX-3).** If declined, the vendor
cannot bid produce at all — a conditional withdrawal expressed as an
exception. A price-delta field cannot hold it.
**Answered-but-negative (Anacostia).** "We are not currently an approved
USDA Foods processor." Squarely answers the requirement; the substance is a
disclosure of inability. Coverage and quality are orthogonal, and a system
that scores this like Potomac's 340,000 lbs is broken.
**Three certifications in one sentence (Chesapeake).** Attachments B, C and
D are satisfied by a single clause naming all three. A per-requirement
extractor must split one span three ways.
## Finding: a gate that no response document can satisfy
`R-B6C` — "Two bid copies submitted" — is a physical submission fact.
Potomac happens to assert it; the other two do not, and no better extraction
would change that. Coverage over it is neither answered nor not_answered but
**unknowable from the artifact**.
Surfaced by `validate-responses.py`, not by design. Treating it as
not_answered would disqualify two of three bidders over a packaging detail.
Raised against #10.