feat(scoring): nightly grant-embedding workflow (Stage 2, step 1)
embedGrants (cron 04:15, after the ingest crons): open grants with a
synopsis and no vector → buildGrantEmbeddingText (pure, tested:
title+funder+program areas+synopsis, 8K cap) → gemini-embedding-001 @
1536 dims RETRIEVAL_DOCUMENT (org profiles will embed as
RETRIEVAL_QUERY on the other side) → grants.synopsis_embedding.
Chunked embed→store (100/chunk) so failures resume from the last
stored chunk; 500/run spend cap.
Core: serverListGrantsNeedingEmbedding (open+unembedded, closest
deadline first), serverSetGrantEmbeddings. Worker gains
@novelpad/outreach-ai dep; wired into main.ts and run-once (incl. the
missed run-once deps injection).
Live-verified: 199/199 open grants embedded in 16s; semantic probe
('after-school STEM education for youth') ranks NCI Youth Enjoy
Science R25 first at 0.639 cosine via the hnsw index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -19,6 +19,10 @@ import { schema } from '@novelpad/outreach-core';
|
||||
import { drizzle } from 'drizzle-orm/node-postgres';
|
||||
import pg from 'pg';
|
||||
|
||||
import {
|
||||
runEmbedGrantsNow,
|
||||
setEmbedGrantsDeps,
|
||||
} from './workflows/embed-grants.js';
|
||||
import { runEnrichOrgsNow, setEnrichOrgsDeps } from './workflows/enrich-orgs.js';
|
||||
import {
|
||||
runExpireGrantsNow,
|
||||
@@ -43,6 +47,7 @@ const RUNNERS: Record<string, () => Promise<void>> = {
|
||||
ingestNhdojOrgs: runIngestNhdojOrgsNow,
|
||||
enrichOrgs: runEnrichOrgsNow,
|
||||
expireGrants: runExpireGrantsNow,
|
||||
embedGrants: runEmbedGrantsNow,
|
||||
};
|
||||
|
||||
const FIRST_RUN_ORDER = [
|
||||
@@ -51,6 +56,7 @@ const FIRST_RUN_ORDER = [
|
||||
'ingestNhdojOrgs',
|
||||
'expireGrants',
|
||||
'enrichOrgs',
|
||||
'embedGrants',
|
||||
];
|
||||
|
||||
if (process.env.DATABASE_URL == null) {
|
||||
@@ -85,6 +91,7 @@ async function main() {
|
||||
setIngestPndRssDeps({ db });
|
||||
setIngestNhdojOrgsDeps({ db });
|
||||
setEnrichOrgsDeps({ db });
|
||||
setEmbedGrantsDeps({ db });
|
||||
|
||||
DBOS.setConfig({
|
||||
name: 'helmdocs-outreach-worker',
|
||||
|
||||
Reference in New Issue
Block a user