Files
grant-outreach-engine/apps/outreach-worker
Croissant Le Doux a532f0bebf feat(scoring): nightly grant-embedding workflow (Stage 2, step 1)
embedGrants (cron 04:15, after the ingest crons): open grants with a
synopsis and no vector → buildGrantEmbeddingText (pure, tested:
title+funder+program areas+synopsis, 8K cap) → gemini-embedding-001 @
1536 dims RETRIEVAL_DOCUMENT (org profiles will embed as
RETRIEVAL_QUERY on the other side) → grants.synopsis_embedding.
Chunked embed→store (100/chunk) so failures resume from the last
stored chunk; 500/run spend cap.

Core: serverListGrantsNeedingEmbedding (open+unembedded, closest
deadline first), serverSetGrantEmbeddings. Worker gains
@novelpad/outreach-ai dep; wired into main.ts and run-once (incl. the
missed run-once deps injection).

Live-verified: 199/199 open grants embedded in 16s; semantic probe
('after-school STEM education for youth') ranks NCI Youth Enjoy
Science R25 first at 0.639 cosine via the hnsw index.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 16:26:41 -04:00
..

@novelpad/outreach-worker

DBOS executor process for the HelmDocs grant-match outreach engine. Runs the scheduled workflows that keep the outreach schema's grant catalog fresh — nightly ingestion from upstream sources and an hourly expiry sweep — decoupled from any HTTP-facing app in this monorepo (mirrors the workflow-worker split in novelpad-desktop).

Boot order (load-bearing)

src/main.ts imports ./workflows/ingest-grants.js and ./workflows/expire-grants.js before calling DBOS.launch(). Each of those modules calls DBOS.registerWorkflow + DBOS.registerScheduled at module-evaluation time — DBOS only dispatches scheduled/queued jobs for functions that were registered before launch, so importing them after launch (or not at all) silently means the cron jobs never fire.

The db handle is then threaded into each workflow module via its set*Deps injector (setIngestGrantsDeps / setExpireGrantsDeps), also before launch — DBOS serializes scheduled-function arguments, so a Drizzle client can't be passed through the scheduler call itself. This is the same module-scope-registry pattern novelpad-desktop's workflow-worker uses for setStartDeps.

1. import workflow modules       → registers ingestGrants / expireGrants
2. build pg Pool + drizzle(db)
3. set*Deps({ db })              → populates each workflow's registry
4. DBOS.setConfig(...)
5. DBOS.launch()                 → scheduler starts firing

Workflows

  • ingestGrants (0 3 * * *, nightly) — fetchGrantsGov (stub; real implementation calls the Grants.gov Search2 API, POST https://api.grants.gov/v1/api/search2) → normalize → upsert via serverInsertGrants from @novelpad/outreach-core/server. NH state postings and 990-PF extracts land as additional fetch+normalize steps later.
  • expireGrants (0 * * * *, hourly) — marks grants whose close date has passed as closed via serverExpireClosedGrants, so they drop out of the active match/scoring pool.

Both are registered as a DBOS workflow and a scheduled function referencing the same function object (DBOS.registerWorkflow then DBOS.registerScheduled) — see the doc comments in src/workflows/*.ts for why the dual registration is required.

Scripts

  • yarn devnode --env-file=.env --env-file-if-exists=.env.local --import tsx/esm ./src/main.ts
  • yarn build — esbuild bundle to build/main.js (--packages=external)
  • yarn start — run the built bundle
  • yarn typechecktsc --noEmit

Environment

Copy .env.example to .env and fill in DATABASE_URL (required — main.ts throws on boot without it) and GCP_SERVICE_ACCOUNT_KEY_PATH (used by future ingestion/scoring steps that call Google-hosted APIs).