feat(ingestion): Phase 1 data spine — Grants.gov, PND RSS, NHDOJ registry, ProPublica enrichment

Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry
(case-insensitive name/city/state key, new-registrant signal),
serverEnrichOrg (IRS fields + re-band, all-null marks attempted),
serverListOrgsNeedingEnrichment.

Worker: four source verticals, each a thin fail-loud client + pure
tested normalize layer + DBOS scheduled workflow:
- ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity
  details (cap 200, logged drops, 250ms politeness) -> batched upsert
- ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/
  deadline extraction, link-keyed upsert
- ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure
  row reconstruction (multi-line names, inferred columns, fail-loud on
  layout change), registry upsert; no-op warn when PDF URL unset
- enrichOrgs (daily): ProPublica search -> conservative name/city match
  (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for
  checkpointed resume, >20% batch failure rethrows

94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Croissant Le Doux
2026-07-16 13:11:50 -04:00
parent 14200edb60
commit 1735ff6754
35 changed files with 3427 additions and 46 deletions

View File

@@ -8,7 +8,8 @@
"dev": "node --env-file=.env --env-file-if-exists=.env.local --import tsx/esm ./src/main.ts",
"build": "esbuild src/main.ts --bundle --platform=node --format=esm --outfile=build/main.js --packages=external",
"start": "node ./build/main.js",
"typecheck": "tsc --noEmit"
"typecheck": "tsc --noEmit",
"test": "vitest run"
},
"imports": {
"#~/*": "./src/*"
@@ -18,6 +19,8 @@
"@dbos-inc/drizzle-datasource": "4.17.6",
"@novelpad/outreach-core": "workspace:^",
"drizzle-orm": "0.44.6",
"fast-xml-parser": "^4.5.0",
"pdfjs-dist": "^4.10.38",
"pg": "8.20.0",
"tsx": "^4.19.2"
},
@@ -26,6 +29,7 @@
"@types/node": "^22",
"@types/pg": "8.20.0",
"esbuild": "^0.24.0",
"typescript": "^5.9.3"
"typescript": "^5.9.3",
"vitest": "^3.2.4"
}
}