Build the synthesized response corpus #30
Notifications
Due Date
No due date set.
Blocks
#37 Build the clickable POC prototype for the internal demo
christian/helmdocs-proposal-system
Reference: christian/helmdocs-proposal-system#30
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Task
Real completed vendor responses proved unobtainable, and v1 is now a demoable POC whose schema is
designed against synthesized responses. This is that corpus.
Write responses to solicitations actually collected in the artifacts research — constrained by
their real requirement lists, real scoring rubrics, and real submission instructions. Not invented
on both sides.
Lightweight but with teeth. The corpus exists to stress the schema, not flatter it, so it must
include:
gradient) and one answering in free prose with no structural anchor (hard end)
has to represent absence
exceptionisfirst-class for
indeterminaterather thanansweredornot_answered, to exercise the three-state coverage modelDeliverable: the corpus committed to this repo, plus a short note on any place the schema failed to
represent something the corpus needed — those are the schema's real findings.
Recorded limitation from the parent ticket: a synthesized corpus reflects the schema author's
assumptions back at them and cannot falsify the design, only exercise it.
Parent: #1
Starting plan
Form settled by author direction: real documents paired with ground-truth label files. The
corpus serves two masters that want different artifacts - #37 needs renderable content, and #16's
completeness sweep needs labelled known-misses or its recall is unmeasurable. Only this form
satisfies both, and the truth files double as test fixtures.
Step zero, and it is decaying
We hold none of the seven source documents. #22 read them from live URLs; nothing was
archived. #25's POC framing rests on writing responses to solicitations we genuinely hold, so
without them the corpus has no foundation.
#22's ninth hazard applies directly: live procurement PDFs returned 403 or bot-challenge HTML six
times during that session, and public-sector URLs rot without notice. Archival is the only
irreversible-if-lost part of this ticket and it contains no decisions. It should happen before
anything else.
Retrieval must also detect "this is not actually a PDF" before storing - a saved challenge page
looks like success until someone tries to parse it.
Anchor order
tabulation (winner 71.81, runner-up 70.89, three vendors). It is therefore the only document
where our ranking arithmetic can be checked against a real outcome rather than against itself.
Everything else is synthesis validating its own assumptions.
sheet lives in a separate Excel attachment, which exercises #22's second extraction hazard.
shortlist-for-interviews stage that exercises the
finalistrelease level from #12.Required stress cases, gathered from three resolutions
Scattered across #25, #16 and #15, collected here so none is missed:
scalar. Must be computable: #15 bars any model fallback, so a shape the code cannot evaluate
has nowhere to go
absence are both represented
completeness sweep ships unexercised
What this will do to the schema
The corpus exists to stress the schema, not flatter it. #25 recorded the accepted risk that a
synthesized corpus reflects its author's assumptions back at them and cannot falsify the design.
Expect the pressure to land on rate structures (#29) and free-prose binding, which #22 flagged as
the two highest-variance areas.
Findings that contradict the schema should be raised as amendments to the relevant closed ticket,
not absorbed silently into the corpus.
Step zero done: all seven source documents archived
Committed as
e7d68aaundercorpus/with aREADME. 3.8 MB. All seven retrieved; none of the 403s #22 hit
recurred - but that was luck rather than durability, and the reason to have done it now.
Correction: Tarrant County cannot anchor the corpus
It is a one-page award tabulation, not a solicitation. My starting plan called it the anchor on
the strength of its real score data; that was wrong on the specific point that matters - it carries
no requirements, so nothing can be written against it.
What it actually holds:
Three vendors inside 1.19 points, tighter than #22 recorded - it noted the top two and there
is a third at 70.62. This remains uniquely valuable, just in a different role: it is the only place
in the corpus where our ranking arithmetic can be wrong in a detectable way rather than
self-consistent. It is also hard evidence for #15's fractional-score decision, since rounding
erases this result completely.
Revised roles
friendship-pcs/rfp.pdfslc-airport/rfp.pdffinalistrelease level from #12.tarrant-county/rfp.pdfreference/*.pdfTwo further corrections to #22's record
set by a wide margin, and a 142-page solicitation is a harder extraction target than the research
implied when it informed #10 and #16.
Next, and a tooling note
Authoring starts on Friendship PCS: three response documents, each deliberately broken in a
specific way, plus truth files. Worth completing that one set as a template before touching SLC, so
the format proves itself once rather than three times.
A real PDF text extractor is needed first. Ad-hoc stream decompression recovers title pages and
little else - it reported zero instances of "shall" across a 52-page solicitation, which is an
artefact of the method rather than the document. Anything measuring extraction against these needs
a proper library, and that choice belongs to whoever opens the extraction seam.
Progress: Friendship PCS extracted, two hazards confirmed
Committed.
corpus/friendship-pcs/extracted/plusNOTES.md.
52pp, 105,154 chars, no page needed OCR - the text layer is clean throughout, so this document
never exercises the OCR ladder. 149 x "shall" and 64 x "must".
The rubric matches #22 exactly and is nested three deep. It also produced a schema finding
raised against #10:
criterionhas no parent, and Tarrant County and Friendship PCS disagree aboutwhich level carries the score.
Two hazards confirmed in the wild
The rubric never uses "criteria" as a heading. It sits under "SECTION K: Evaluation Factors" ->
"K.2 Technical Rating Scale", with weights as bullet suffixes. Searching for "evaluation criteria"
or "scoring" returns nothing - direct evidence that #16's sweep cannot lean on headings.
Schedule C is the price sheet and contains no prices. Its entire body:
Pricing is 40 of 100 points on this solicitation and a PDF-only extractor recovers none of it.
This is a ready-made labelled known-miss: the obligation is present, the content is not, and no
better PDF parsing recovers it. First entry for the truth file.
Tooling note, resolved
The earlier "we need a real PDF extractor" note is closed - see the reframe on #19. For authoring,
plain pdfjs is enough and free; for anything measured, the Document AI path is the one that
produces spans.
Friendship PCS response set complete
Committed
19e7ffe. Three responses inresponses/source/with paired truth files, plus
validate-responses.py.All nine stress cases from #25, #15 and #16 are covered and the validator asserts it, so the
checklist cannot rot silently.
The field has a real outcome
The lowest bid is disqualified and the highest is strongest on the criterion the SFA wrote into
its own rubric. Deliberate: a ranking that cannot represent absence puts Chesapeake first, and a
cost comparison that ignores the school calendar flatters Anacostia. Both failure modes are now
detectable rather than theoretical.
Traps planted, each aimed at a specific decision
$589,570; the document says $589,970. #15 decision 2 put price arithmetic in code so a stated
total is never trusted over a computed one - this is the case that catches an extractor lifting
the vendor's figure.
computed total is possible at all. The KM-1 shape reaching the response side.
ones with deltas (Anacostia) as the control.
A conditional withdrawal wearing an exception's clothes - a
price_deltafield cannot hold it,which is a constraint on #29.
squarely answers the requirement and discloses inability. Coverage and quality are orthogonal.
Second schema finding raised
R-B6Cis a mandatory gate no response document can establish. Raised against #10.Next
SLC Airport - 142pp, tabbed A-M binding, and the document that exercises the OCR ladder Friendship
never touches. Its shortlist-for-interviews stage is also the only artifact that reaches the
finalistrelease level.Resolution
Demo-sufficient, and closed on that basis rather than on exhausting the plan.
The corpus set for Friendship PCS is complete: the archived
solicitation, hand-authored ground truth, three synthesized responses with paired truth files, and
two validators. Committed across
e7d68aa,0b64eda,e81da40,19e7ffe,126e94a.All nine stress cases from #25, #15 and #16 are covered, and
validate-responses.pyasserts thechecklist so it cannot rot silently.
It did the job it existed to do
#25 accepted the risk that a synthesized corpus "reflects the schema author's assumptions back at
them and cannot falsify the design." That was the right caution and it was too pessimistic. The
set produced two schema changes, both now applied to the layout:
criterioncould not represent a nested rubric. Friendship's weighted children sum exactlyto their parents while two other criteria carry unweighted guidance - both shapes in one
document. Forced the scoring-level rule.
design, which is the argument for having written one.
Neither was predictable from reading the RFP research. Both came from the friction of authoring
against a real artifact.
What is deliberately not here
SLC Airport, Tarrant responses, and any OCR exercise. Friendship's text layer is clean
throughout, so the corpus never touches the OCR ladder, and Tarrant is an award tabulation with no
requirements to answer.
That is a real gap and it belongs to whoever opens the extraction seam, not to the demo:
represented only by a synthesized matrix, not a real one
finalistrelease level - SLC's shortlist-for-interviews stage is the only artifact thatreaches it
MAX(1/12 x minimum guarantee, % of gross receipts), a shape noneof the three synthesized rate structures cover
Recorded in Not yet specified rather than left implicit.