Add three synthesized Friendship PCS responses with ground truth
Covers all nine stress cases from #25, #15 and #16. The lowest bid is disqualified on two missing health documents and the highest bid is strongest on the SFA's own geographic-preference criterion, so a ranking that cannot represent absence gets the field exactly backwards. Deliberate traps: a planted $400 arithmetic error in a stated total, an unverifiable total whose unit price lives only in the external Excel, an unlabelled exception, a non-priceable exception that is really a conditional withdrawal, and an answered-but-negative response. Second schema finding, surfaced by the validator: R-B6C is a mandatory gate that no response document can establish. Coverage over it is unknowable rather than absent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -63,3 +63,62 @@ present, its content is not, and no amount of better PDF parsing recovers it.
|
||||
|
||||
149 × "shall", 64 × "must" across 52 pages. An earlier ad-hoc extraction reported
|
||||
zero of each; that was the extractor failing, not the document.
|
||||
|
||||
---
|
||||
|
||||
# Response set
|
||||
|
||||
Three synthesized responses in `responses/source/`, each paired with a truth
|
||||
file. Designed so the field has a real outcome rather than three
|
||||
interchangeable bids.
|
||||
|
||||
| Vendor | Binding | Price | Gates | Distinguishing feature |
|
||||
|---|---|---|---|---|
|
||||
| Potomac Provisions | tabbed compliance matrix | $589,970 | PASS | Complete; volume-banded pricing; 18% local |
|
||||
| Chesapeake Food Service | free prose, no anchors | $541,280 | **FAIL** | Cheapest and disqualified; unlabelled exception |
|
||||
| Anacostia Farm Collective | named sections + enumeration | $624,315 | PASS | 61% local; three labelled exceptions; seasonal pricing |
|
||||
|
||||
The lowest bid is disqualified and the highest bid is strongest on the
|
||||
criterion the SFA wrote into its own rubric. That is deliberate: a ranking
|
||||
that cannot represent absence puts Chesapeake first.
|
||||
|
||||
## Deliberate traps
|
||||
|
||||
**Planted arithmetic error (Potomac).** The stated base-year total is
|
||||
$589,970; band price times scenario volume is $589,570. A $400 gap. #15
|
||||
decision 2 puts all price arithmetic in code precisely so a stated total is
|
||||
never trusted over a computed one — this is the case that catches an
|
||||
extractor lifting the vendor's own figure.
|
||||
|
||||
**Unverifiable total (Chesapeake).** The case price exists only in the
|
||||
external Excel sheet, so no computed total can be produced at all. Contrast
|
||||
Potomac, where the arithmetic can be checked and fails.
|
||||
|
||||
**Unlabelled exception (Chesapeake).** A price-relief request phrased as
|
||||
politeness inside the Pricing section, with no heading and no delta.
|
||||
Anacostia's three exceptions are explicitly headed with deltas and act as
|
||||
the control.
|
||||
|
||||
**Non-priceable exception (Anacostia, EX-3).** If declined, the vendor
|
||||
cannot bid produce at all — a conditional withdrawal expressed as an
|
||||
exception. A price-delta field cannot hold it.
|
||||
|
||||
**Answered-but-negative (Anacostia).** "We are not currently an approved
|
||||
USDA Foods processor." Squarely answers the requirement; the substance is a
|
||||
disclosure of inability. Coverage and quality are orthogonal, and a system
|
||||
that scores this like Potomac's 340,000 lbs is broken.
|
||||
|
||||
**Three certifications in one sentence (Chesapeake).** Attachments B, C and
|
||||
D are satisfied by a single clause naming all three. A per-requirement
|
||||
extractor must split one span three ways.
|
||||
|
||||
## Finding: a gate that no response document can satisfy
|
||||
|
||||
`R-B6C` — "Two bid copies submitted" — is a physical submission fact.
|
||||
Potomac happens to assert it; the other two do not, and no better extraction
|
||||
would change that. Coverage over it is neither answered nor not_answered but
|
||||
**unknowable from the artifact**.
|
||||
|
||||
Surfaced by `validate-responses.py`, not by design. Treating it as
|
||||
not_answered would disqualify two of three bidders over a packaging detail.
|
||||
Raised against #10.
|
||||
|
||||
Reference in New Issue
Block a user