Apply both corpus findings to the schema
criterion.parent_id — a score commits at any criterion with NO WEIGHTED CHILDREN. Derived from the rubric, never configured, and neither corpus document is special-cased: Tarrant's five flat criteria are all leaves; Friendship's 'Method of Approach' 30 is not scoreable while its 15/10/5 children are. A criterion whose sub-items carry no points stays scoreable at its own level, because those are requirements mapped through criterion_requirement rather than children. requirement.determinable_from_response — false where satisfaction cannot be established from the response document at all. Gate evaluation excludes these rather than failing them; treating Friendship's 'two bid copies' as not_answered disqualifies two of three bidders over a packaging detail. The scoring rule is mirrored in corpus/validate-truth.py so the corpus and the code cannot silently disagree. Against Friendship it yields 10 scoreable criteria of 12, summing to exactly 100. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -53,6 +53,19 @@ if pts != list(range(t['ratingScale']['min'], t['ratingScale']['max'] + 1)):
|
||||
else:
|
||||
print(f" ok rating scale {pts[0]}-{pts[-1]}, {len(pts)} anchors")
|
||||
|
||||
# The scoring-level rule, mirrored from src/scoring/routing.ts: a score commits
|
||||
# at any criterion with NO WEIGHTED CHILDREN. Printed so the corpus and the code
|
||||
# cannot silently disagree about which criteria a committee actually scores.
|
||||
scoreable = [c for c in t['criteria']
|
||||
if not any(k['parentId'] == c['id'] and (k.get('maxPoints') or 0) > 0
|
||||
for k in t['criteria'])]
|
||||
sc_total = sum(c['maxPoints'] for c in scoreable)
|
||||
print(f" ok scoreable criteria {len(scoreable)} of {len(t['criteria'])}, "
|
||||
f"points sum {sc_total}")
|
||||
if sc_total != tot:
|
||||
errs.append(f"scoreable points sum {sc_total} != declared total {tot}")
|
||||
print(" " + ", ".join(f"{c['id']}({c['maxPoints']})" for c in scoreable))
|
||||
|
||||
mapped = {c for r in t['requirements'] for c in r.get('criterionIds', [])}
|
||||
for cid, c in crit.items():
|
||||
kids = [x for x in t['criteria'] if x['parentId'] == cid]
|
||||
|
||||
Reference in New Issue
Block a user