feat(ingestion): Phase 1 data spine — Grants.gov, PND RSS, NHDOJ registry, ProPublica enrichment

Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry
(case-insensitive name/city/state key, new-registrant signal),
serverEnrichOrg (IRS fields + re-band, all-null marks attempted),
serverListOrgsNeedingEnrichment.

Worker: four source verticals, each a thin fail-loud client + pure
tested normalize layer + DBOS scheduled workflow:
- ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity
  details (cap 200, logged drops, 250ms politeness) -> batched upsert
- ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/
  deadline extraction, link-keyed upsert
- ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure
  row reconstruction (multi-line names, inferred columns, fail-loud on
  layout change), registry upsert; no-op warn when PDF URL unset
- enrichOrgs (daily): ProPublica search -> conservative name/city match
  (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for
  checkpointed resume, >20% batch failure rethrows

94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Croissant Le Doux
2026-07-16 13:11:50 -04:00
parent 14200edb60
commit 1735ff6754
35 changed files with 3427 additions and 46 deletions

View File

@@ -0,0 +1,26 @@
/**
* HTTP client for the Philanthropy News Digest "RFPs" RSS feed.
*
* Kept deliberately thin — a plain `fetch` returning the raw XML body.
* Parsing/normalization lives in `normalize.ts` so it can be unit tested
* against fixture strings without a network call.
*/
/** Default PND RFP feed URL; overridable via `PND_RFP_FEED_URL` at the call site. */
export const PND_RFP_FEED_URL = 'https://philanthropynewsdigest.org/feeds/rfps.rss';
/**
* Fetches the raw RSS XML body from the PND RFP feed.
*
* Throws on any non-2xx response so the caller's step-level retry policy
* (see `ingest-pnd-rss.ts`) can kick in.
*/
export async function fetchPndRfpFeed(feedUrl: string): Promise<string> {
const response = await fetch(feedUrl);
if (!response.ok) {
throw new Error(
`fetchPndRfpFeed: ${feedUrl} responded with ${response.status} ${response.statusText}`,
);
}
return response.text();
}