Files
helmdocs-proposal-system/corpus/friendship-pcs/responses/chesapeake-food-service.truth.json
Christian 19e7ffe92a Add three synthesized Friendship PCS responses with ground truth
Covers all nine stress cases from #25, #15 and #16. The lowest bid is
disqualified on two missing health documents and the highest bid is
strongest on the SFA's own geographic-preference criterion, so a ranking
that cannot represent absence gets the field exactly backwards.

Deliberate traps: a planted $400 arithmetic error in a stated total, an
unverifiable total whose unit price lives only in the external Excel, an
unlabelled exception, a non-priceable exception that is really a conditional
withdrawal, and an answered-but-negative response.

Second schema finding, surfaced by the validator: R-B6C is a mandatory gate
that no response document can establish. Coverage over it is unknowable
rather than absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 18:39:08 -04:00

238 lines
9.7 KiB
JSON

{
"$comment": "Ground truth for source/chesapeake-food-service.md. The hard end of the binding gradient.",
"vendor": {
"id": "CHESAPEAKE",
"name": "Chesapeake Food Service, Inc.",
"location": "Baltimore, MD"
},
"bindingMechanism": {
"level": "free_prose",
"$comment": "#22's least reliable tier. Four sections exist but carry no requirement-level anchors, no RFP references and no matrix. Recovering which sentence answers which requirement is text classification, not structural lookup. This is the document that will expose extraction weakness."
},
"expectedOutcome": {
"gatesPassed": false,
"failedGates": [
"R-B6D",
"R-C5-INSPECTIONS"
],
"atRiskGates": [
"R-B2-LOCAL",
"R-C1-VALUES",
"R-A2"
],
"narrative": "Lowest price in the field at $541,280 and DISQUALIFIED regardless, on two missing mandatory health documents. This is the case that proves absence must be representable — a ranking that scores it on price alone puts a disqualified bid at the top."
},
"answers": [
{
"requirementId": "R-A1",
"coverage": "answered",
"evidence": "Cover block present"
},
{
"requirementId": "R-A2",
"coverage": "not_answered",
"evidence": null,
"$comment": "No checklist is reproduced anywhere."
},
{
"requirementId": "R-B6D",
"coverage": "not_answered",
"evidence": null,
"isGateFailure": true,
"$comment": "THE PRIMARY OMISSION. A current state or local health certificate is never mentioned. Not refused, not deferred — simply absent. The response lists many enclosures, which makes the absence easy for a human to miss and is exactly why coverage must be computed per requirement rather than eyeballed."
},
{
"requirementId": "R-C5-INSPECTIONS",
"coverage": "not_answered",
"evidence": null,
"isGateFailure": true,
"$comment": "Two most recent health inspections never mentioned. The response DOES discuss pest management at the same facility, which is topically adjacent and will tempt a weak extractor into a false positive."
},
{
"requirementId": "R-B2-LOCAL",
"coverage": "indeterminate",
"evidence": "'Chesapeake buys from the Maryland and Delmarva growing region in season... We would expect a meaningful proportion of produce delivered between June and October to originate within the region.'",
"$comment": "Addresses the TOPIC warmly and satisfies none of the OBLIGATION — no percentage, no certification, no acceptance of the audit right. This is KM-2's requirement meeting a response that gestures at it. The hardest coverage call in the corpus."
},
{
"requirementId": "R-C1-VALUES",
"coverage": "indeterminate",
"evidence": "'We are committed to student nutrition and to the communities we serve' — one clause in the Executive Summary, under no heading.",
"$comment": "Exercises KM-3 from the response side. Whether this constitutes a 'response to values statement' is a judgement, not a lookup."
},
{
"requirementId": "R-B6C",
"coverage": "not_answered",
"evidence": null
},
{
"requirementId": "R-B7-FORMAT",
"coverage": "answered",
"evidence": "Four sections present",
"$comment": "Section three is titled 'Method of Approach and Implementation', dropping 'Plan'. Near-miss on a required section title — a strict matcher fails, a fuzzy one passes."
},
{
"requirementId": "R-B8-QUEST",
"coverage": "answered",
"evidence": "'Three references are provided in the questionnaire' — implies completion, never states it"
},
{
"requirementId": "R-B8-LICENSE",
"coverage": "answered",
"evidence": "'Our business license and certificate of occupancy are enclosed'"
},
{
"requirementId": "R-B8-OCCUPANCY",
"coverage": "answered",
"evidence": "Same sentence as the business license"
},
{
"requirementId": "R-B8-FINANCIALS",
"coverage": "answered",
"evidence": "Two years audited, Hanlon & Reeve LLP"
},
{
"requirementId": "R-D6-INSURANCE",
"coverage": "answered",
"evidence": "$2M per occurrence / $4M aggregate, FPCS as additional insured"
},
{
"requirementId": "R-ATT-B",
"coverage": "answered",
"evidence": "'debarment, price determination and lobbying certifications are signed and enclosed'"
},
{
"requirementId": "R-ATT-C",
"coverage": "answered",
"evidence": "Same sentence"
},
{
"requirementId": "R-ATT-D",
"coverage": "answered",
"evidence": "Same sentence",
"$comment": "THREE separate certification requirements satisfied by ONE sentence naming all three. A per-requirement extractor must split one span three ways."
},
{
"requirementId": "R-E1-EEO",
"coverage": "answered",
"evidence": "'as are the equal opportunity and clean air and water certifications'"
},
{
"requirementId": "R-E3-CLEANAIR",
"coverage": "answered",
"evidence": "Same sentence"
},
{
"requirementId": "R-SCHED-C",
"coverage": "answered",
"evidence": "'submitted separately on the Excel schedule provided'"
},
{
"requirementId": "R-USDA-FOODS",
"coverage": "answered",
"evidence": "'We participate in the USDA Foods programme in Maryland... our processing relationships are stronger in the protein categories than in fruit and vegetable.'",
"$comment": "ANSWERED but WEAK. Coverage and quality are orthogonal — a correct extraction marks this answered and a correct score marks it low. Conflating them is a defect."
},
{
"requirementId": "R-CN-LABELS",
"coverage": "answered",
"evidence": "'category managers hold CN labels and formulation statements... typically within two business days of a request'"
},
{
"requirementId": "R-C6-PEST",
"coverage": "answered",
"evidence": "'licensed applicator, serviced monthly'"
},
{
"requirementId": "R-D1-DELIVERY",
"coverage": "answered",
"evidence": "Site survey of all eight locations; delivery window confirmed with authorised designee"
},
{
"requirementId": "R-B5-PRICING-TERMS",
"coverage": "answered",
"evidence": "'Prices are held firm through the base year'"
},
{
"requirementId": "R-D8-EPA",
"coverage": "answered",
"evidence": "'relief... in the event of a commodity movement exceeding eight percent'"
},
{
"requirementId": "R-C5-FOODSAFETY",
"coverage": "indeterminate",
"evidence": "Pest management is described; no GFSI, SQF or equivalent certification is claimed.",
"$comment": "Partial. The requirement asks for a food safety programme and the response supplies one component of one."
},
{
"requirementId": "R-ATT-A",
"coverage": "indeterminate",
"evidence": "'the District's Healthy Schools Act obligations' named in passing, no compliance detail"
},
{
"requirementId": "R-C7-QA",
"coverage": "not_answered",
"evidence": null
},
{
"requirementId": "R-C8-FARM2SCHOOL",
"coverage": "not_answered",
"evidence": null
},
{
"requirementId": "R-C4-MENUS",
"coverage": "not_answered",
"evidence": null
},
{
"requirementId": "R-D3-RECORDS",
"coverage": "not_answered",
"evidence": null
}
],
"priceQuote": {
"rateStructure": {
"kind": "flat_unit_plus_frequency_surcharge",
"$comment": "Exercises #29 differently from Potomac. Cost depends on a variable the SFA controls per site and may change mid-term with two weeks' notice — so derived_cost is only defined against a declared frequency assumption, and the response states one.",
"unit": "case",
"flatUnitPrice": null,
"$note": "Unit case price lives only in the external Excel sheet — the KM-1 shape again, this time making the vendor's own stated total unverifiable from the document alone.",
"perStopCharge": [
{
"deliveriesPerWeek": 2,
"chargePerStop": 34.0
},
{
"deliveriesPerWeek": 3,
"chargePerStop": 28.5
},
{
"deliveriesPerWeek": 4,
"chargePerStop": 24.0
}
],
"frequencyChangeableWithNotice": "2 weeks"
},
"statedBaseYearTotal": 541280,
"statedAssumptions": {
"deliveriesPerWeek": 3,
"sites": 8,
"calendarDays": 180
},
"$verify": "UNVERIFIABLE from this document. The case price is absent, so no computed total can be produced — only the vendor's stated figure exists. Contrast Potomac, where the arithmetic can be checked and fails."
},
"exceptions": [
{
"id": "IMPLICIT-1",
"labelled": false,
"requirementId": "R-B5-PRICING-TERMS",
"text": "We would ask only for relief under the economic price adjustment provision in the event of a commodity movement exceeding eight percent",
"$comment": "AN UNLABELLED EXCEPTION. It is phrased as a polite request inside the Pricing section, carries no heading, no price delta and no 'exception' wording. Compare Anacostia, which labels three exceptions explicitly with deltas. #22 called exceptions the thing evaluators care most about and the thing an invented corpus always forgets — this is the version that is present but hidden."
}
],
"extractionDifficulty": {
"expectRecall": "low",
"$comment": "No anchors of any kind. Expect false positives on R-C5-INSPECTIONS from adjacent pest-management prose, and expect the three-certifications-in-one-sentence span to be split incorrectly or attributed to only one requirement."
}
}