Files
grant-outreach-engine/packages/outreach-core
Croissant Le Doux 1735ff6754 feat(ingestion): Phase 1 data spine — Grants.gov, PND RSS, NHDOJ registry, ProPublica enrichment
Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry
(case-insensitive name/city/state key, new-registrant signal),
serverEnrichOrg (IRS fields + re-band, all-null marks attempted),
serverListOrgsNeedingEnrichment.

Worker: four source verticals, each a thin fail-loud client + pure
tested normalize layer + DBOS scheduled workflow:
- ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity
  details (cap 200, logged drops, 250ms politeness) -> batched upsert
- ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/
  deadline extraction, link-keyed upsert
- ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure
  row reconstruction (multi-line names, inferred columns, fail-loud on
  layout change), registry upsert; no-op warn when PDF URL unset
- enrichOrgs (daily): ProPublica search -> conservative name/city match
  (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for
  checkpointed resume, >20% batch failure rethrows

94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 13:11:50 -04:00
..

@novelpad/outreach-core

Domain/data package for the HelmDocs grant-match outreach engine: the Drizzle schema for grants, orgs, org profiles, contacts, matches, and pipeline events, plus the actions/queries that read and write them.

Modeled on packages/core in novelpad-desktop, following the same action/query pattern: every read lives in a domain's queries/, every write in its actions/, both barrel-exported, and route handlers/workflows/hooks never touch db.select()/db.insert() directly.

Domains

  • grants/ — open grant opportunities (Grants.gov, NH state, 990-PF, PND RSS, Candid). Ingestion upserts on sourceUrl; an hourly sweep expires anything past its close date.
  • orgs/ — NH nonprofits, plus org_profiles (LLM/agent-researched enrichment: mission, programs, known funders, budget band, citation sources).
  • matches/ — a scored (org, grant) pair. hard-gates.ts is the deterministic, DB-free pass/fail predicate (entity eligibility, geography, ≥21-day runway, ≥$10k ceiling, application-form support) that runs before the weighted LLM subscoring pass — see its module doc for why it's kept pure.
  • pipeline/ — outreach lifecycle events (enrolledsentopened/replied/... → demo_bookedconverted), synced from Apollo and recorded internally.

Deliberate deviations from @novelpad/core

This package is server-only — there is no client app, so several things @novelpad/core does for its PGlite/OPFS client don't apply here:

  • No PGlite, no client schema, no sync. There's exactly one database handle type (NpOutreachDatabase/NpOutreachTransaction in src/db/db.ts), not the client/server pair (NpClientDatabase vs NpDatabase) @novelpad/core needs to support both a local and a remote copy of the same data.
  • No novelpad_core_dev export condition. @novelpad/core's package.json uses a custom novelpad_core_dev condition so dev builds resolve straight to src/*.ts without a build step. This package's exports map only has plain types/default conditions — run yarn build (or yarn dev for a watch build) before consuming it from a sibling workspace in dev.
  • No build:migration-files bundling step. @novelpad/core's create:migration also runs a script that bundles generated SQL into a TS module the client can import for its OPFS-based local migrator. There is no local migrator here, so bin/create-migration only runs drizzle-kit generate against the single server config.
  • Root index.ts is intentionally small. It exports only the schema, DB types, and the pure hard-gates predicate — not domain client barrels, since every domain here is server-only. Everything else lives behind index.server.ts / the ./server export condition.

Scripts

yarn build            # tsc
yarn dev              # tsc --watch
yarn typecheck        # tsc --noEmit
yarn test             # vitest run
yarn create:migration <name>   # drizzle-kit generate against drizzle.config.ts
yarn migrate           # drizzle-kit migrate
yarn studio             # drizzle-kit studio