feat(ingestion): Phase 1 data spine — Grants.gov, PND RSS, NHDOJ registry, ProPublica enrichment
Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry (case-insensitive name/city/state key, new-registrant signal), serverEnrichOrg (IRS fields + re-band, all-null marks attempted), serverListOrgsNeedingEnrichment. Worker: four source verticals, each a thin fail-loud client + pure tested normalize layer + DBOS scheduled workflow: - ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity details (cap 200, logged drops, 250ms politeness) -> batched upsert - ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/ deadline extraction, link-keyed upsert - ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure row reconstruction (multi-line names, inferred columns, fail-loud on layout change), registry upsert; no-op warn when PDF URL unset - enrichOrgs (daily): ProPublica search -> conservative name/city match (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for checkpointed resume, >20% batch failure rethrows 94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
26
apps/outreach-worker/src/sources/pnd-rss/client.ts
Normal file
26
apps/outreach-worker/src/sources/pnd-rss/client.ts
Normal file
@@ -0,0 +1,26 @@
|
||||
/**
|
||||
* HTTP client for the Philanthropy News Digest "RFPs" RSS feed.
|
||||
*
|
||||
* Kept deliberately thin — a plain `fetch` returning the raw XML body.
|
||||
* Parsing/normalization lives in `normalize.ts` so it can be unit tested
|
||||
* against fixture strings without a network call.
|
||||
*/
|
||||
|
||||
/** Default PND RFP feed URL; overridable via `PND_RFP_FEED_URL` at the call site. */
|
||||
export const PND_RFP_FEED_URL = 'https://philanthropynewsdigest.org/feeds/rfps.rss';
|
||||
|
||||
/**
|
||||
* Fetches the raw RSS XML body from the PND RFP feed.
|
||||
*
|
||||
* Throws on any non-2xx response so the caller's step-level retry policy
|
||||
* (see `ingest-pnd-rss.ts`) can kick in.
|
||||
*/
|
||||
export async function fetchPndRfpFeed(feedUrl: string): Promise<string> {
|
||||
const response = await fetch(feedUrl);
|
||||
if (!response.ok) {
|
||||
throw new Error(
|
||||
`fetchPndRfpFeed: ${feedUrl} responded with ${response.status} ${response.statusText}`,
|
||||
);
|
||||
}
|
||||
return response.text();
|
||||
}
|
||||
Reference in New Issue
Block a user