feat(scoring): nightly grant-embedding workflow (Stage 2, step 1)

embedGrants (cron 04:15, after the ingest crons): open grants with a
synopsis and no vector → buildGrantEmbeddingText (pure, tested:
title+funder+program areas+synopsis, 8K cap) → gemini-embedding-001 @
1536 dims RETRIEVAL_DOCUMENT (org profiles will embed as
RETRIEVAL_QUERY on the other side) → grants.synopsis_embedding.
Chunked embed→store (100/chunk) so failures resume from the last
stored chunk; 500/run spend cap.

Core: serverListGrantsNeedingEmbedding (open+unembedded, closest
deadline first), serverSetGrantEmbeddings. Worker gains
@novelpad/outreach-ai dep; wired into main.ts and run-once (incl. the
missed run-once deps injection).

Live-verified: 199/199 open grants embedded in 16s; semantic probe
('after-school STEM education for youth') ranks NCI Youth Enjoy
Science R25 first at 0.639 cosine via the hnsw index.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Croissant Le Doux
2026-07-16 16:26:41 -04:00
parent 8d57d57557
commit a532f0bebf
13 changed files with 329 additions and 1 deletions

View File

@@ -0,0 +1,39 @@
import { describe, expect, it } from 'vitest';
import { buildGrantEmbeddingText } from './grant-embedding-text.js';
describe('buildGrantEmbeddingText', () => {
it('composes title, funder, program areas, and synopsis', () => {
const text = buildGrantEmbeddingText({
funder: 'National Science Foundation',
title: 'STEM Education Grants',
synopsis: 'Funding for after-school STEM programs.',
programAreas: ['B25', 'O50'],
});
expect(text).toBe(
'STEM Education Grants — National Science Foundation.\n' +
'Program areas: B25, O50.\n' +
'Funding for after-school STEM programs.',
);
});
it('omits empty program areas and blank synopsis', () => {
const text = buildGrantEmbeddingText({
funder: 'F',
title: 'T',
synopsis: ' ',
programAreas: [],
});
expect(text).toBe('T — F.');
});
it('truncates very long synopses', () => {
const text = buildGrantEmbeddingText({
funder: 'F',
title: 'T',
synopsis: 'x'.repeat(20_000),
programAreas: null,
});
expect(text.length).toBe(8_000);
});
});

View File

@@ -0,0 +1,35 @@
/**
* Composes the text that represents a grant in embedding space. Pure —
* shared by the nightly embed job and any ad-hoc re-embedding so a grant
* is always embedded from identically-composed text.
*
* Kept intentionally simple: funder + title + program areas + synopsis,
* truncated to stay well inside the embedding model's context. Mission-fit
* retrieval compares org-profile text against this, so the composition
* should read like a description, not a key-value dump.
*/
export interface GrantEmbeddingInput {
readonly funder: string;
readonly title: string;
readonly synopsis: string | null;
readonly programAreas: readonly string[] | null;
}
/** ~8K chars ≈ well under gemini-embedding-001's 2048-token input limit
* for typical English prose after the API's own truncation; the tail of a
* long federal synopsis is boilerplate anyway. */
const MAX_TEXT_LENGTH = 8_000;
export function buildGrantEmbeddingText(grant: GrantEmbeddingInput): string {
const parts: string[] = [`${grant.title}${grant.funder}.`];
if (grant.programAreas != null && grant.programAreas.length > 0) {
parts.push(`Program areas: ${grant.programAreas.join(', ')}.`);
}
if (grant.synopsis != null && grant.synopsis.trim() !== '') {
parts.push(grant.synopsis.trim());
}
return parts.join('\n').slice(0, MAX_TEXT_LENGTH);
}

View File

@@ -5,3 +5,4 @@ export * from './agents/org-profiler/schema.js';
export * from './agents/org-profiler/run.js';
export * from './agents/mission-fit-judge/schema.js';
export * from './agents/mission-fit-judge/run.js';
export * from './grant-embedding-text.js';