feat(scoring): nightly grant-embedding workflow (Stage 2, step 1)

embedGrants (cron 04:15, after the ingest crons): open grants with a
synopsis and no vector → buildGrantEmbeddingText (pure, tested:
title+funder+program areas+synopsis, 8K cap) → gemini-embedding-001 @
1536 dims RETRIEVAL_DOCUMENT (org profiles will embed as
RETRIEVAL_QUERY on the other side) → grants.synopsis_embedding.
Chunked embed→store (100/chunk) so failures resume from the last
stored chunk; 500/run spend cap.

Core: serverListGrantsNeedingEmbedding (open+unembedded, closest
deadline first), serverSetGrantEmbeddings. Worker gains
@novelpad/outreach-ai dep; wired into main.ts and run-once (incl. the
missed run-once deps injection).

Live-verified: 199/199 open grants embedded in 16s; semantic probe
('after-school STEM education for youth') ranks NCI Youth Enjoy
Science R25 first at 0.639 cosine via the hnsw index.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Croissant Le Doux
2026-07-16 16:26:41 -04:00
parent 8d57d57557
commit a532f0bebf
13 changed files with 329 additions and 1 deletions

View File

@@ -17,6 +17,7 @@
"dependencies": {
"@dbos-inc/dbos-sdk": "4.17.6",
"@dbos-inc/drizzle-datasource": "4.17.6",
"@novelpad/outreach-ai": "workspace:^",
"@novelpad/outreach-core": "workspace:^",
"drizzle-orm": "0.44.6",
"fast-xml-parser": "^4.5.0",