1735ff6754
Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry (case-insensitive name/city/state key, new-registrant signal), serverEnrichOrg (IRS fields + re-band, all-null marks attempted), serverListOrgsNeedingEnrichment. Worker: four source verticals, each a thin fail-loud client + pure tested normalize layer + DBOS scheduled workflow: - ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity details (cap 200, logged drops, 250ms politeness) -> batched upsert - ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/ deadline extraction, link-keyed upsert - ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure row reconstruction (multi-line names, inferred columns, fail-loud on layout change), registry upsert; no-op warn when PDF URL unset - enrichOrgs (daily): ProPublica search -> conservative name/city match (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for checkpointed resume, >20% batch failure rethrows 94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 lines
921 B
TypeScript
27 lines
921 B
TypeScript
/**
|
|
* HTTP client for the Philanthropy News Digest "RFPs" RSS feed.
|
|
*
|
|
* Kept deliberately thin — a plain `fetch` returning the raw XML body.
|
|
* Parsing/normalization lives in `normalize.ts` so it can be unit tested
|
|
* against fixture strings without a network call.
|
|
*/
|
|
|
|
/** Default PND RFP feed URL; overridable via `PND_RFP_FEED_URL` at the call site. */
|
|
export const PND_RFP_FEED_URL = 'https://philanthropynewsdigest.org/feeds/rfps.rss';
|
|
|
|
/**
|
|
* Fetches the raw RSS XML body from the PND RFP feed.
|
|
*
|
|
* Throws on any non-2xx response so the caller's step-level retry policy
|
|
* (see `ingest-pnd-rss.ts`) can kick in.
|
|
*/
|
|
export async function fetchPndRfpFeed(feedUrl: string): Promise<string> {
|
|
const response = await fetch(feedUrl);
|
|
if (!response.ok) {
|
|
throw new Error(
|
|
`fetchPndRfpFeed: ${feedUrl} responded with ${response.status} ${response.statusText}`,
|
|
);
|
|
}
|
|
return response.text();
|
|
}
|