christian/helmdocs-proposal-system
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

Define the common schema for RFPs and responses #10

New Issue
Closed
opened 2026-08-02 03:05:52 +00:00 by christian · 5 comments
christian commented 2026-08-02 03:05:52 +00:00
Owner
Copy Link

Question

The load-bearing domain model for the whole system. What is a requirement? What is a rubric criterion? What is a response element, and how does it bind to the requirement it answers?

Both sides submit documents and both get extracted, so this schema is the target of two different extractors and the substrate for ranking, matching, analytics, and the cross-retailer profile. It has to be expressive enough to hold what real retail RFPs contain — pricing tables, narrative, attachments, certifications, volume tiers — and rigid enough that a hundred vendors' responses are genuinely comparable.

Prior art to mine, not copy: packages/core/src/solicitation-extraction/schema.ts and the compliance-matrix-builder agent in the novelpad-desktop repo.

Resolve: the entity model, the requirement<->response binding, how retailer-specific fields coexist with the common core, and what is deliberately left unstructured.

Parent: #1

## Question The load-bearing domain model for the whole system. What is a *requirement*? What is a *rubric criterion*? What is a *response element*, and how does it bind to the requirement it answers? Both sides submit documents and both get extracted, so this schema is the target of two different extractors and the substrate for ranking, matching, analytics, and the cross-retailer profile. It has to be expressive enough to hold what real retail RFPs contain — pricing tables, narrative, attachments, certifications, volume tiers — and rigid enough that a hundred vendors' responses are genuinely comparable. Prior art to mine, not copy: `packages/core/src/solicitation-extraction/schema.ts` and the compliance-matrix-builder agent in the `novelpad-desktop` repo. Resolve: the entity model, the requirement<->response binding, how retailer-specific fields coexist with the common core, and what is deliberately left unstructured. --- Parent: #1
christian added the
wayfinder:grilling
wayfinder:ticket
 labels 2026-08-02 03:05:52 +00:00
christian added a new dependency 2026-08-02 03:06:25 +00:00
#16 Design the extraction confirmation step on both sides
christian added a new dependency 2026-08-02 03:06:25 +00:00
#15 Design AI bid ranking, scoring, and its defensibility
christian added a new dependency 2026-08-02 03:06:25 +00:00
#14 Specify the two-party data release rules
christian added a new dependency 2026-08-02 03:06:26 +00:00
#19 Draw the HelmDocs integration boundary
christian added a new dependency 2026-08-02 03:06:26 +00:00
#18 Define the retailer integration and export paths
christian added a new dependency 2026-08-02 03:11:53 +00:00
#23 Design the app framework — retailer and vendor surfaces
christian added a new dependency 2026-08-02 03:56:02 +00:00
#26 Define the category taxonomy
christian self-assigned this 2026-08-02 05:03:44 +00:00
christian commented 2026-08-02 05:17:48 +00:00
Author
Owner
Copy Link

Resolution

Four decisions, then the model they imply. Sized for a demoable POC per the amended destination —
lightweight, but with teeth in the places where a naive schema quietly produces wrong awards.

1. Requirement is the atom; criteria are a weighted overlay

Real solicitations carry two structures that do not line up. A requirement list — what a
vendor must do, provide, or comply with. And an evaluation rubric — weighted criteria the bid is
scored against. They are authored by different people for different purposes, and they are not in
1:1 correspondence: one criterion ("Technical Approach, 40 pts") spans a dozen requirements, while
many requirements ("provide a current certificate of insurance") carry no score at all and are
pure pass/fail gates.

So: an RFP decomposes into requirements, each with a kind — mandatory / scored /
informational. Criteria are a separate weighted structure, each mapping to a set of
requirements. A response binds answers to requirement ids; ranking rolls those answers up
through the mapping to criteria.

Why not make the criterion the atom. Mandatory-but-unscored requirements would have nowhere to
live — and those are precisely the ones that disqualify a bid. A vendor scoring 95 who never
supplied a W-9 would rank first, with the disqualification that actually decides the award invisible
to the system.

Why not the section or question. Section structure is document formatting, not meaning, and it
differs on every RFP. Binding to it would optimise for the one thing that never needs to be
portable, and cross-RFP comparability, answer reuse, and matching would all lose their footing.

This also matches what the existing compliance-matrix-builder agent already does, so the
bidder-side thinking transfers rather than being re-derived.

What it costs. Two structures plus a mapping is the most schema of the three options, and
authoring the mapping is real work on every RFP — a step that can be wrong.

2. The document is legally authoritative; the confirmed extraction is authoritative for evaluation

Retailers arrive with a finished RFP document — settled by the distribution-layer decision — so the
requirement list and criterion mapping must come out of that document. But in real procurement the
document is the binding artifact: if the extracted list diverges from the PDF and an award is
protested, the PDF wins.

So extraction proposes; the retailer confirms; the confirmed structure is what ranking,
matching, and compliance checking run on, while the document remains binding. Confirmation is a
recorded event with an actor and a timestamp.

Teeth: every requirement carries a provenance pointer into the source document — document
id, page, character span — plus a confirmation state. Any ranking claim therefore traces back to a
location a human can look at.

Why this over native authoring. Authoring requirements natively would give perfect data and no
divergence, but it contradicts the distribution-layer decision and asks a retailer to move authoring
off a process built around legal review and templates — the largest adoption ask available, made to
the side that pays nothing.

Why this over advisory-only annotations. Nothing would be comparable across RFPs, ranking would
degrade to per-document summarisation, and the cross-retailer premise would have no structure to
stand on.

What it costs. Confirmation is friction on the retailer, who is the free side and least tolerant
of work. And nothing guarantees completeness — a requirement missed entirely by extraction never
appears for anyone to confirm. That is an unfixed hole, and it is the strongest argument for the
extraction confirmation surface being good rather than adequate.

3. Coverage is three-state, and the vendor attests to their own

Binding is a gradient — from a tabbed compliance matrix restating every requirement verbatim,
down to free prose answering three requirements in one paragraph while naming none. So "no answer
bound to this requirement" is ambiguous between the vendor did not answer (which can disqualify)
and we failed to find it (which is our defect). Collapsing those disqualifies vendors for
extraction bugs.

Each requirement therefore carries a coverage state per response:

  • answered — bound, with a provenance span in the vendor's document
  • not_answered — extraction searched and is confident nothing addresses it
  • indeterminate — binding unreliable

Only not_answered may drive disqualification. indeterminate always escalates, never
penalises.

The vendor resolves their own indeterminates at submit — the same move as the vault decision:
the vendor attests, we validate structure. A bid therefore arrives with coverage the vendor has
stood behind, and a later dispute is between the vendor and their own attestation rather than
between the retailer and our extractor.

What it costs. Three states propagate everywhere downstream — ranking, the compliance matrix UI,
the audit record. And the confidence line between not_answered and indeterminate is a tuning
parameter that will be wrong at first; it should be observable and adjustable rather than baked in.

4. Price is a rate function evaluated against an RFP-declared scenario

Real bids do not price as a number. They price as unit rates, volume bands, tiers, minimums, and
surcharges — and two vendors can each be cheaper than the other depending on how much is actually
bought. Comparability is not a property of the bid; it exists only relative to an assumed volume.

Which means: whoever picks the volume assumption picks the winner.

So a price line is a typed rate structure, never a scalar. The RFP declares an evaluation
scenario — the volume basket price will be judged against — as part of the solicitation, so
vendors know the basis before they bid. Comparable cost is computed at evaluation time from rate
function plus scenario, and stored as a derived value carrying the scenario id it came from.

Why declared up front rather than chosen at evaluation. Choosing the basket after prices are
known makes it an unpublished lever applied to known bids — the exact pattern a losing vendor
protests, and not a position this product should be standing in. Published up front, the winner is a
function of a stated assumption, and re-running against a different scenario is a visible act
rather than a quiet re-rank.

Why not leave price unmodelled. Price is usually the heaviest-weighted criterion in the rubric.
An unmodelled price means the ranking engine abstains on the thing that decides the award, which
undercuts the central claim that responses are machine-comparable.

What it costs. Retailers must state a volume basket up front, and some genuinely do not know it.
The rate-structure type is the most intricate object in the schema and will need extending —
graduated to its own ticket.

Carried forward unchanged from the artifacts research

Scoring scales are per-document and are never normalised. Scores are stored as raw
points / max_points against the criterion that issued them. There is no cross-RFP score and no
global vendor rating derived from scores. A 92 on one solicitation and an 88 on another are not
comparable and the schema must not offer a field that implies they are.

The model

Lightweight but with teeth. Roughly fourteen entities.

Solicitation side

solicitation           id, issuer_org_id, local_category_id, title, stage,
                       issued_at, acknowledgement_required
solicitation_document  id, solicitation_id, kind, blob_ref, sha256      -- legally authoritative
requirement            id, solicitation_id, ordinal, kind(mandatory|scored|informational),
                       text, response_obligation,
                       provenance(document_id, page, span),
                       confirmation_state, confirmed_by, confirmed_at
criterion              id, solicitation_id, label, max_points, weight
criterion_requirement  criterion_id, requirement_id                     -- the overlay mapping
price_line             id, solicitation_id, label, unit
evaluation_scenario    id, solicitation_id, declared_at                 -- published with the RFP
scenario_quantity      scenario_id, price_line_id, quantity

Response side

response               id, solicitation_id, vendor_org_id, submitted_at,
                       attested_by, attested_at
response_document      id, response_id, blob_ref, sha256
answer                 id, response_id, requirement_id,
                       coverage(answered|not_answered|indeterminate),
                       text, provenance(document_id, page, span),
                       binding_confidence, vendor_attested
exception              id, response_id, requirement_id?, text, provenance
price_quote            id, response_id, price_line_id, rate_structure
derived_cost           id, response_id, scenario_id, amount, computed_at
score                  id, response_id, criterion_id, points, max_points  -- raw, never normalised

exception is deliberately first-class. Exceptions and qualifications taken against the retailer's
terms are what real evaluators care about most, and they are the thing an invented corpus always
forgets.

Deliberately absent: any cross-RFP score, any normalised rating, any scalar total price, any
requirement hierarchy, and any structure keyed to document sections.

Category reference: solicitation.local_category_id points at the retailer-local category
per the taxonomy decision — not the spine leaf. The spine is reached through the mapping.

Constraints handed to other tickets

  • Data release: answer.text, price_quote.rate_structure, and exception.text are the
    competitively sensitive fields; release rules key off (response, requirement) granularity, not
    whole documents.
  • Ranking: rolls raw scores through criterion_requirement; must represent indeterminate
    rather than scoring it; consumes derived_cost, never rate_structure directly.
  • Extraction confirmation: covers both the retailer confirming requirements and the vendor
    resolving indeterminate coverage. The completeness hole in decision 2 is that surface's problem.
  • Audit: confirmation events, vendor attestation, and derived_cost.scenario_id are all
    protest surface.
  • App framework: unblocked. The compliance matrix is the central retailer surface, and coverage
    state is its primary visual.
  • The rate structure type and the synthesized response corpus graduated to their own
    tickets.
## Resolution Four decisions, then the model they imply. Sized for a demoable POC per the amended destination — lightweight, but with teeth in the places where a naive schema quietly produces wrong awards. ### 1. Requirement is the atom; criteria are a weighted overlay Real solicitations carry **two structures that do not line up**. A **requirement list** — what a vendor must do, provide, or comply with. And an **evaluation rubric** — weighted criteria the bid is scored against. They are authored by different people for different purposes, and they are not in 1:1 correspondence: one criterion ("Technical Approach, 40 pts") spans a dozen requirements, while many requirements ("provide a current certificate of insurance") carry **no score at all** and are pure pass/fail gates. So: an RFP decomposes into **requirements**, each with a kind — `mandatory` / `scored` / `informational`. **Criteria** are a separate weighted structure, each mapping to a *set* of requirements. A response binds answers to **requirement ids**; ranking rolls those answers up through the mapping to criteria. **Why not make the criterion the atom.** Mandatory-but-unscored requirements would have nowhere to live — and those are precisely the ones that disqualify a bid. A vendor scoring 95 who never supplied a W-9 would rank first, with the disqualification that actually decides the award invisible to the system. **Why not the section or question.** Section structure is document formatting, not meaning, and it differs on every RFP. Binding to it would optimise for the one thing that never needs to be portable, and cross-RFP comparability, answer reuse, and matching would all lose their footing. This also matches what the existing `compliance-matrix-builder` agent already does, so the bidder-side thinking transfers rather than being re-derived. **What it costs.** Two structures plus a mapping is the most schema of the three options, and authoring the mapping is real work on every RFP — a step that can be wrong. ### 2. The document is legally authoritative; the confirmed extraction is authoritative for evaluation Retailers arrive with a finished RFP document — settled by the distribution-layer decision — so the requirement list and criterion mapping must come out of that document. But **in real procurement the document is the binding artifact**: if the extracted list diverges from the PDF and an award is protested, the PDF wins. So extraction *proposes*; the retailer *confirms*; the **confirmed structure is what ranking, matching, and compliance checking run on**, while the document remains binding. Confirmation is a recorded event with an actor and a timestamp. **Teeth:** every requirement carries a **provenance pointer into the source document** — document id, page, character span — plus a confirmation state. Any ranking claim therefore traces back to a location a human can look at. **Why this over native authoring.** Authoring requirements natively would give perfect data and no divergence, but it contradicts the distribution-layer decision and asks a retailer to move authoring off a process built around legal review and templates — the largest adoption ask available, made to the side that pays nothing. **Why this over advisory-only annotations.** Nothing would be comparable across RFPs, ranking would degrade to per-document summarisation, and the cross-retailer premise would have no structure to stand on. **What it costs.** Confirmation is friction on the retailer, who is the free side and least tolerant of work. And **nothing guarantees completeness** — a requirement missed entirely by extraction never appears for anyone to confirm. That is an unfixed hole, and it is the strongest argument for the extraction confirmation surface being good rather than adequate. ### 3. Coverage is three-state, and the vendor attests to their own Binding is a **gradient** — from a tabbed compliance matrix restating every requirement verbatim, down to free prose answering three requirements in one paragraph while naming none. So "no answer bound to this requirement" is ambiguous between **the vendor did not answer** (which can disqualify) and **we failed to find it** (which is our defect). Collapsing those disqualifies vendors for extraction bugs. Each requirement therefore carries a **coverage state per response**: - `answered` — bound, with a provenance span in the vendor's document - `not_answered` — extraction searched and is confident nothing addresses it - `indeterminate` — binding unreliable **Only `not_answered` may drive disqualification. `indeterminate` always escalates, never penalises.** **The vendor resolves their own indeterminates at submit** — the same move as the vault decision: the vendor attests, we validate structure. A bid therefore arrives with coverage **the vendor has stood behind**, and a later dispute is between the vendor and their own attestation rather than between the retailer and our extractor. **What it costs.** Three states propagate everywhere downstream — ranking, the compliance matrix UI, the audit record. And the confidence line between `not_answered` and `indeterminate` is a tuning parameter that will be wrong at first; it should be observable and adjustable rather than baked in. ### 4. Price is a rate function evaluated against an RFP-declared scenario Real bids do not price as a number. They price as **unit rates, volume bands, tiers, minimums, and surcharges** — and two vendors can each be cheaper than the other depending on how much is actually bought. **Comparability is not a property of the bid; it exists only relative to an assumed volume.** Which means: **whoever picks the volume assumption picks the winner.** So a price line is a **typed rate structure, never a scalar**. The **RFP declares an evaluation scenario** — the volume basket price will be judged against — as part of the solicitation, so vendors know the basis before they bid. Comparable cost is **computed at evaluation time** from rate function plus scenario, and stored as a derived value **carrying the scenario id it came from**. **Why declared up front rather than chosen at evaluation.** Choosing the basket after prices are known makes it an unpublished lever applied to known bids — the exact pattern a losing vendor protests, and not a position this product should be standing in. Published up front, the winner is a function of a stated assumption, and re-running against a different scenario is a **visible act** rather than a quiet re-rank. **Why not leave price unmodelled.** Price is usually the heaviest-weighted criterion in the rubric. An unmodelled price means the ranking engine abstains on the thing that decides the award, which undercuts the central claim that responses are machine-comparable. **What it costs.** Retailers must state a volume basket up front, and some genuinely do not know it. The rate-structure type is the most intricate object in the schema and will need extending — graduated to its own ticket. ### Carried forward unchanged from the artifacts research **Scoring scales are per-document and are never normalised.** Scores are stored as raw `points / max_points` against the criterion that issued them. There is **no cross-RFP score** and no global vendor rating derived from scores. A 92 on one solicitation and an 88 on another are not comparable and the schema must not offer a field that implies they are. ### The model Lightweight but with teeth. Roughly fourteen entities. **Solicitation side** ``` solicitation id, issuer_org_id, local_category_id, title, stage, issued_at, acknowledgement_required solicitation_document id, solicitation_id, kind, blob_ref, sha256 -- legally authoritative requirement id, solicitation_id, ordinal, kind(mandatory|scored|informational), text, response_obligation, provenance(document_id, page, span), confirmation_state, confirmed_by, confirmed_at criterion id, solicitation_id, label, max_points, weight criterion_requirement criterion_id, requirement_id -- the overlay mapping price_line id, solicitation_id, label, unit evaluation_scenario id, solicitation_id, declared_at -- published with the RFP scenario_quantity scenario_id, price_line_id, quantity ``` **Response side** ``` response id, solicitation_id, vendor_org_id, submitted_at, attested_by, attested_at response_document id, response_id, blob_ref, sha256 answer id, response_id, requirement_id, coverage(answered|not_answered|indeterminate), text, provenance(document_id, page, span), binding_confidence, vendor_attested exception id, response_id, requirement_id?, text, provenance price_quote id, response_id, price_line_id, rate_structure derived_cost id, response_id, scenario_id, amount, computed_at score id, response_id, criterion_id, points, max_points -- raw, never normalised ``` `exception` is deliberately first-class. Exceptions and qualifications taken against the retailer's terms are what real evaluators care about most, and they are the thing an invented corpus always forgets. **Deliberately absent:** any cross-RFP score, any normalised rating, any scalar total price, any requirement hierarchy, and any structure keyed to document sections. **Category reference:** `solicitation.local_category_id` points at the **retailer-local** category per the taxonomy decision — not the spine leaf. The spine is reached through the mapping. ### Constraints handed to other tickets - **Data release**: `answer.text`, `price_quote.rate_structure`, and `exception.text` are the competitively sensitive fields; release rules key off `(response, requirement)` granularity, not whole documents. - **Ranking**: rolls raw scores through `criterion_requirement`; must represent `indeterminate` rather than scoring it; consumes `derived_cost`, never `rate_structure` directly. - **Extraction confirmation**: covers both the retailer confirming requirements and the vendor resolving indeterminate coverage. The completeness hole in decision 2 is that surface's problem. - **Audit**: confirmation events, vendor attestation, and `derived_cost.scenario_id` are all protest surface. - **App framework**: unblocked. The compliance matrix is the central retailer surface, and coverage state is its primary visual. - **The rate structure type** and **the synthesized response corpus** graduated to their own tickets.
christian commented 2026-08-02 15:22:59 +00:00
Author
Owner
Copy Link

Two notes from a skeptical review pass

1. criterion_requirement is authored, not extracted, and has no provenance. Every other object
in this model traces to a document span. Real solicitations almost never state which requirements
roll up to which criterion — the mapping is implicit. So decision 1 asks the retailer not to
confirm an extraction but to create information the binding document does not contain, and
decision 2's "the document is legally authoritative" has nothing to be authoritative over here.

Decision 1 called the mapping "real work on every RFP", which understates it: it is the one place
the system manufactures structure. Under protest, why was this requirement scored under Technical
Approach answers only "a retailer clicked it".

Consequences to carry: the mapping needs its own attribution (who mapped, when, and whether the
model proposed it), and ranking explanations must visibly distinguish document-derived from
human-authored structure. Handed to the extraction confirmation step and
ranking.

2. The evaluation scenario is the most commercially sensitive field in the solicitation.
Decision 4 publishes the volume basket with the RFP so price is bindable up front.
Visibility names volume commitments as exactly what the acknowledgement gate protects, and
that gate is optional per RFP — so with it off, the basket reaches the entire roster.

Not a contradiction, but the release timing is unspecified and neither ticket noticed the
interaction. Handed to data release.

## Two notes from a skeptical review pass **1. `criterion_requirement` is authored, not extracted, and has no provenance.** Every other object in this model traces to a document span. Real solicitations almost never state which requirements roll up to which criterion — the mapping is implicit. So decision 1 asks the retailer not to *confirm an extraction* but to **create information the binding document does not contain**, and decision 2's "the document is legally authoritative" has nothing to be authoritative over here. Decision 1 called the mapping "real work on every RFP", which understates it: it is the one place the system manufactures structure. Under protest, *why was this requirement scored under Technical Approach* answers only "a retailer clicked it". Consequences to carry: the mapping needs its own attribution (who mapped, when, and whether the model proposed it), and ranking explanations must **visibly distinguish document-derived from human-authored structure**. Handed to [the extraction confirmation step](https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/16) and [ranking](https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15). **2. The evaluation scenario is the most commercially sensitive field in the solicitation.** Decision 4 publishes the volume basket with the RFP so price is bindable up front. [Visibility](https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/12) names volume commitments as exactly what the acknowledgement gate protects, and that gate is **optional per RFP** — so with it off, the basket reaches the entire roster. Not a contradiction, but the release timing is unspecified and neither ticket noticed the interaction. Handed to [data release](https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/14).
christian commented 2026-08-03 22:19:00 +00:00
Author
Owner
Copy Link

Corpus finding: criterion cannot represent a nested rubric

Raised as an amendment rather than absorbed silently, per #30's plan. Found while extracting
Friendship PCS.

The schema has criterion (id, solicitation_id, label, max_points, weight) with no parent. Two
real documents in the corpus disagree about which level carries the score, and the schema can only
express one of them.

Tarrant County scores at the top level - five flat criteria, no children:

Management & Support Staff 25 | Meal and Service Plan 25 | Financial Stability 20
Meal Price 25 | References 5

Friendship PCS has weighted sub-criteria that sum to their parent:

Method of Approach and Implementation            30
  Food and Packaging                             15
  Utilization of USDA Foods                      10
  Geographic preference                           5

Flattening to the leaves loses the parent label the document itself names, and loses the fact that
30 points is the unit a committee may score against. Keeping only the parent loses the weighting.

Suggested shape: a nullable parent_criterion_id, plus an explicit rule for which level scores
are committed at - because #15's commit model needs to know, and #16's confirmation ledger counts
"criterion-mapped" requirements against it.

Not a gap, and worth recording as validated: the same document also has sub-bullets that carry
no points - under Proven Organizational Capacity and Community Connection. Those are guidance, not
sub-criteria, and criterion_requirement already models them correctly. The design holds; only
weighted nesting is missing.

Second, smaller finding. Friendship's "Geographic preference - 5 points" embeds a mandatory
submission inside a scored criterion: "Bidder must submit certification of percentage of locally
grown or raised foods." A scored criterion carrying a pass/fail artifact. #15's routing decides by
requirement kind, so this needs to be two requirements mapped to one criterion rather than one
requirement of ambiguous kind.

## Corpus finding: `criterion` cannot represent a nested rubric Raised as an amendment rather than absorbed silently, per https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/30's plan. Found while extracting [Friendship PCS](https://gitea.stephenmann.io/christian/helmdocs-proposal-system/src/branch/master/corpus/friendship-pcs/NOTES.md). **The schema has `criterion (id, solicitation_id, label, max_points, weight)` with no parent.** Two real documents in the corpus disagree about which level carries the score, and the schema can only express one of them. **Tarrant County** scores at the top level - five flat criteria, no children: ``` Management & Support Staff 25 | Meal and Service Plan 25 | Financial Stability 20 Meal Price 25 | References 5 ``` **Friendship PCS** has weighted sub-criteria that sum to their parent: ``` Method of Approach and Implementation 30 Food and Packaging 15 Utilization of USDA Foods 10 Geographic preference 5 ``` Flattening to the leaves loses the parent label the document itself names, and loses the fact that 30 points is the unit a committee may score against. Keeping only the parent loses the weighting. **Suggested shape:** a nullable `parent_criterion_id`, plus an explicit rule for which level scores are committed at - because https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15's commit model needs to know, and https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/16's confirmation ledger counts "criterion-mapped" requirements against it. **Not a gap, and worth recording as validated:** the same document also has sub-bullets that carry *no* points - under Proven Organizational Capacity and Community Connection. Those are guidance, not sub-criteria, and `criterion_requirement` already models them correctly. The design holds; only weighted nesting is missing. **Second, smaller finding.** Friendship's "Geographic preference - 5 points" embeds a mandatory submission inside a scored criterion: *"Bidder must submit certification of percentage of locally grown or raised foods."* A scored criterion carrying a pass/fail artifact. https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15's routing decides by requirement kind, so this needs to be two requirements mapped to one criterion rather than one requirement of ambiguous kind.
christian commented 2026-08-03 22:39:42 +00:00
Author
Owner
Copy Link

Second corpus finding: a mandatory gate no response document can satisfy

Surfaced by validate-responses.py while checking the Friendship PCS response set - by the
validator, not by design, which is the point of having one.

R-B6C - "Two bid copies submitted" is on the Bidder Checklist and is therefore a mandatory
gate. It is also a physical submission fact. One of three synthesized bidders happens to assert
it in prose; the other two simply do not, and no better extraction would change that.

Coverage over this requirement is neither answered nor not_answered. It is unknowable from the
artifact. #10's three-state coverage has no cell for it, and the nearest available state is
actively wrong: treating it as not_answered disqualifies two of three bidders over a packaging
detail.

This is not a rare shape. Anything about the act of submitting rather than the content of the
submission behaves the same way - copy counts, delivery method, envelope marking, receipt by a
stated hour. Real solicitations are full of them and they are genuinely mandatory.

Suggested shape: a determinable_from_response flag on requirement, so gate evaluation can
exclude these rather than fail them. It also matters to #16 - a vendor should not be shown "you
will fail a gate" for something their document was never able to prove - and to #15, whose gate
routing is code and would otherwise return a confident wrong answer.

The corpus records it as requirementNotes.gatesNotDeterminableFromResponse, and the validator
excludes those from its gate check, so the shape is already exercised.

## Second corpus finding: a mandatory gate no response document can satisfy Surfaced by `validate-responses.py` while checking the Friendship PCS response set - by the validator, not by design, which is the point of having one. **`R-B6C` - "Two bid copies submitted"** is on the Bidder Checklist and is therefore a mandatory gate. It is also a **physical submission fact**. One of three synthesized bidders happens to assert it in prose; the other two simply do not, and no better extraction would change that. Coverage over this requirement is neither `answered` nor `not_answered`. It is **unknowable from the artifact**. https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/10's three-state coverage has no cell for it, and the nearest available state is actively wrong: treating it as `not_answered` disqualifies two of three bidders over a packaging detail. **This is not a rare shape.** Anything about the act of submitting rather than the content of the submission behaves the same way - copy counts, delivery method, envelope marking, receipt by a stated hour. Real solicitations are full of them and they are genuinely mandatory. **Suggested shape:** a `determinable_from_response` flag on `requirement`, so gate evaluation can exclude these rather than fail them. It also matters to https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/16 - a vendor should not be shown "you will fail a gate" for something their document was never able to prove - and to https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15, whose gate routing is `code` and would otherwise return a confident wrong answer. The corpus records it as `requirementNotes.gatesNotDeterminableFromResponse`, and the validator excludes those from its gate check, so the shape is already exercised.
christian commented 2026-08-03 22:51:08 +00:00
Author
Owner
Copy Link

Amendment: the scoring level, and two schema columns

Both corpus findings are now applied - commit 126e94a.

The scoring level, decided

A score commits at any criterion with NO WEIGHTED CHILDREN.

Derived from the rubric, never configured, and neither corpus document needs special-casing:

  • Tarrant County - five flat criteria, no nesting. All five are leaves.
  • Friendship PCS - "Method of Approach" 30 is not scoreable; its children (15/10/5) are.
    Likewise "Experience" 20 over 5/5/5/5.

A criterion whose sub-items carry no points stays scoreable at its own level, because those
sub-items are requirements mapped through criterion_requirement rather than children. Friendship
carries both shapes in one document, which is why the line is drawn on weight rather than on
nesting - identical visual indentation, two different meanings.

Why not score at the parent. It discards weights the issuer published, and #15 has to commit
against something the document actually declares.

Why not make it configurable. The rubric already answers it. A per-solicitation setting would
let a retailer score at a level the document does not support, and an award turns on it.

Consistency is structural rather than enforced: weighted children sum to their parent's
max_points. Against Friendship the rule yields 10 scoreable criteria of 12, summing to exactly
100 - corpus/validate-truth.py asserts it, so the corpus and the code cannot silently drift.

Columns added

criterion.parent_id                    uuid, nullable, self-referencing
requirement.determinable_from_response boolean, default true

determinable_from_response is the second finding: R-B6C "two bid copies" is a mandatory gate no
response document can establish. Gate evaluation excludes these rather than failing them -
treating it as not_answered disqualifies two of three synthesized bidders over a packaging detail.
Confirmation is unaffected; the retailer still confirms the requirement exists.

Handed on: gateEvaluableFromResponse() and scoreableCriteria() in src/scoring/routing.ts.
#16 must also never warn a vendor they will fail a gate their document could not have proven.

## Amendment: the scoring level, and two schema columns Both corpus findings are now applied - commit `126e94a`. ### The scoring level, decided **A score commits at any criterion with NO WEIGHTED CHILDREN.** Derived from the rubric, never configured, and neither corpus document needs special-casing: - **Tarrant County** - five flat criteria, no nesting. All five are leaves. - **Friendship PCS** - "Method of Approach" 30 is **not** scoreable; its children (15/10/5) are. Likewise "Experience" 20 over 5/5/5/5. A criterion whose sub-items carry **no points** stays scoreable at its own level, because those sub-items are requirements mapped through `criterion_requirement` rather than children. Friendship carries **both shapes in one document**, which is why the line is drawn on *weight* rather than on nesting - identical visual indentation, two different meanings. **Why not score at the parent.** It discards weights the issuer published, and https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/15 has to commit against something the document actually declares. **Why not make it configurable.** The rubric already answers it. A per-solicitation setting would let a retailer score at a level the document does not support, and an award turns on it. **Consistency is structural rather than enforced:** weighted children sum to their parent's `max_points`. Against Friendship the rule yields **10 scoreable criteria of 12, summing to exactly 100** - `corpus/validate-truth.py` asserts it, so the corpus and the code cannot silently drift. ### Columns added ``` criterion.parent_id uuid, nullable, self-referencing requirement.determinable_from_response boolean, default true ``` `determinable_from_response` is the second finding: `R-B6C` "two bid copies" is a mandatory gate no response document can establish. **Gate evaluation excludes these rather than failing them** - treating it as `not_answered` disqualifies two of three synthesized bidders over a packaging detail. Confirmation is unaffected; the retailer still confirms the requirement exists. Handed on: `gateEvaluableFromResponse()` and `scoreableCriteria()` in `src/scoring/routing.ts`. https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/16 must also never warn a vendor they will fail a gate their document could not have proven.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
HITL: resolved by conversation
wayfinder:map
The shared map for a wayfinding effort
wayfinder:prototype
HITL: resolved by making something rough
wayfinder:research
AFK: resolved by a research subagent
wayfinder:task
Manual work that unblocks a decision
wayfinder:ticket
A wayfinder decision ticket
No Label
wayfinder:grilling
wayfinder:ticket
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
christian
No Assignees christian
1 Participants
Notifications
Due Date
No due date set.
Blocks
#14 Specify the two-party data release rules
christian/helmdocs-proposal-system
#16 Design the extraction confirmation step on both sides
christian/helmdocs-proposal-system
#15 Design AI bid ranking, scoring, and its defensibility
christian/helmdocs-proposal-system
#19 Draw the HelmDocs integration boundary
christian/helmdocs-proposal-system
#18 Define the retailer integration and export paths
christian/helmdocs-proposal-system
#23 Design the app framework — retailer and vendor surfaces
christian/helmdocs-proposal-system
Depends on
#25 Obtain real completed vendor responses, including a compliance matrix
christian/helmdocs-proposal-system
#26 Define the category taxonomy
christian/helmdocs-proposal-system
Reference: christian/helmdocs-proposal-system#10
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?