1735ff6754
Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry (case-insensitive name/city/state key, new-registrant signal), serverEnrichOrg (IRS fields + re-band, all-null marks attempted), serverListOrgsNeedingEnrichment. Worker: four source verticals, each a thin fail-loud client + pure tested normalize layer + DBOS scheduled workflow: - ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity details (cap 200, logged drops, 250ms politeness) -> batched upsert - ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/ deadline extraction, link-keyed upsert - ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure row reconstruction (multi-line names, inferred columns, fail-loud on layout change), registry upsert; no-op warn when PDF URL unset - enrichOrgs (daily): ProPublica search -> conservative name/city match (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for checkpointed resume, >20% batch failure rethrows 94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@novelpad/outreach-worker
DBOS executor process for the HelmDocs grant-match outreach engine. Runs the
scheduled workflows that keep the outreach schema's grant catalog fresh —
nightly ingestion from upstream sources and an hourly expiry sweep — decoupled
from any HTTP-facing app in this monorepo (mirrors the workflow-worker split
in novelpad-desktop).
Boot order (load-bearing)
src/main.ts imports ./workflows/ingest-grants.js and
./workflows/expire-grants.js before calling DBOS.launch(). Each of
those modules calls DBOS.registerWorkflow + DBOS.registerScheduled at
module-evaluation time — DBOS only dispatches scheduled/queued jobs for
functions that were registered before launch, so importing them after launch
(or not at all) silently means the cron jobs never fire.
The db handle is then threaded into each workflow module via its
set*Deps injector (setIngestGrantsDeps / setExpireGrantsDeps), also
before launch — DBOS serializes scheduled-function arguments, so a Drizzle
client can't be passed through the scheduler call itself. This is the same
module-scope-registry pattern novelpad-desktop's workflow-worker uses for
setStartDeps.
1. import workflow modules → registers ingestGrants / expireGrants
2. build pg Pool + drizzle(db)
3. set*Deps({ db }) → populates each workflow's registry
4. DBOS.setConfig(...)
5. DBOS.launch() → scheduler starts firing
Workflows
ingestGrants(0 3 * * *, nightly) —fetchGrantsGov(stub; real implementation calls the Grants.gov Search2 API,POST https://api.grants.gov/v1/api/search2) →normalize→ upsert viaserverInsertGrantsfrom@novelpad/outreach-core/server. NH state postings and 990-PF extracts land as additional fetch+normalize steps later.expireGrants(0 * * * *, hourly) — marks grants whose close date has passed as closed viaserverExpireClosedGrants, so they drop out of the active match/scoring pool.
Both are registered as a DBOS workflow and a scheduled function referencing
the same function object (DBOS.registerWorkflow then DBOS.registerScheduled)
— see the doc comments in src/workflows/*.ts for why the dual registration
is required.
Scripts
yarn dev—node --env-file=.env --env-file-if-exists=.env.local --import tsx/esm ./src/main.tsyarn build— esbuild bundle tobuild/main.js(--packages=external)yarn start— run the built bundleyarn typecheck—tsc --noEmit
Environment
Copy .env.example to .env and fill in DATABASE_URL (required — main.ts
throws on boot without it) and GCP_SERVICE_ACCOUNT_KEY_PATH (used by
future ingestion/scoring steps that call Google-hosted APIs).