grant-outreach-engine/docs/features/ingestion.md

Data-spine ingestion (Phase 1)

How grant opportunities and NH nonprofit orgs enter the outreach database. All ingestion runs as DBOS scheduled workflows in apps/outreach-worker; every source has a thin impure client, a pure normalize layer (unit-tested on fixtures), and a workflow that upserts through @novelpad/outreach-core/server actions — never inline DB calls.

Grant upserts key on grants.source_url; org registry upserts key on case-insensitive (name, city, state); IRS enrichment applies to org rows by id and re-bands ICP via icpBandForRevenue ($100K–$5M = primary).

Schedules

Workflow	Cron (UTC)	Source
ingestGrants	0 3 * * *	Grants.gov Search2
ingestPndRss	30 3 * * *	Philanthropy News Digest RFP feed
expireGrants	0 * * * *	(sweep: status='expired' past close_date)
enrichOrgs	0 5 * * *	ProPublica Nonprofit Explorer
embedGrants	15 4 * * *	gemini-embedding-001 over open-grant synopses → grants.synopsis_embedding (first paid AI call; ~pennies/batch)
ingestNhdojOrgs	0 4 1 * *	NHDOJ Charitable Trusts registry PDF

Sources

Grants.gov

Source: apps/outreach-worker/src/sources/grants-gov/ — client.ts (fetch), normalize.ts (pure transform), pagination.ts (pure pagination arithmetic). Wired into apps/outreach-worker/src/workflows/ingest-grants.ts.

Cadence: Nightly, 0 3 * * * (03:00 UTC), registered as a DBOS scheduled workflow (ingestGrants, ExactlyOncePerInterval).

API: Public Grants.gov Search2 API, no key required.

• POST https://api.grants.gov/v1/api/search2 — enumerates opportunities, filtered to oppStatuses: 'posted' and nonprofit eligibility codes 12|13 (501(c)(3) and non-501(c)(3) nonprofits). Paginated via rows/startRecordNum.
• POST https://api.grants.gov/v1/api/fetchOpportunity — full detail (synopsis: description, applicant types, award amounts, response date) for a single opportunity id.
• Both endpoints wrap responses in an { errorcode, msg, data } envelope. Any non-2xx HTTP status or errorcode !== 0 throws immediately with a descriptive message — the client never silently drops or swallows an upstream failure.

Caps (per nightly run):

• Search enumeration: up to 1000 hits (SEARCH_HIT_CAP). Count logged via console.log.
• Detail fetch: up to 200 opportunities (DETAIL_FETCH_CAP), each a separate fetchOpportunity call with a ~250ms politeness delay between requests. When the search result set exceeds the cap, the excess is dropped for that run (picked up on a later run) and a console.warn states exactly how many opportunities were skipped — never a silent truncation.
• Upsert: batched in groups of 100 via serverInsertGrants.

Normalization rules (normalizeGrantsGovOpportunity, pure — no I/O):

• sourceUrl (the upsert key): https://www.grants.gov/search-results-detail/{opportunityId} — stable across re-crawls.
• funder: agencyDetails.agencyName → synopsis.agencyName → search hit's agency → an agency code fallback → literal 'Unknown federal agency' if nothing is present.
• synopsis: HTML-stripped from synopsis.synopsisDesc (block tags become newlines, common entities decoded); null if empty after stripping.
• awardFloor / awardCeiling / expectedAwardsCount: parsed from string-or-number wire values (handles "$1,500,000"-style formatting), rounded to whole dollars/counts; null on unparseable input.
• openDate / closeDate: parsed from the search hit's MM/DD/YYYY fields (falling back to the detail's postingDate/responseDate); any invalid, out-of-range, or calendar-rollover date (e.g. 02/30/2026) resolves to null rather than a garbage Date.
• eligibilityEntityTypes: trimmed, non-empty descriptions from synopsis.applicantTypes; null if none.
• geographicScope: always null — federal grants are national by default and Grants.gov has no reliable field to derive a narrower scope from; left unset rather than guessed.
• programAreas: null (not yet derived from this source).
• applicationEffortEstimate: 'unknown'; matchRequirement / applicationFormSupported: false; status: 'open'; source: 'grants_gov'; lastVerifiedAt: set to the normalization timestamp.

Philanthropy News Digest RSS

Status (2026-07-16): feed retired upstream. Every historical feed path (/feeds/rfps.rss and variants) now returns the site's HTML shell — PND is a Next.js app with client-side data and no <link rel=alternate> feeds. The workflow runs and upserts zero rows. Rework candidate: drive their internal JSON API / __NEXT_DATA__ instead. Low priority — PND was "cheap incremental coverage"; Grants.gov + NH state sources carry the corpus.

Nightly ingestion of the Philanthropy News Digest "RFPs" RSS feed (https://philanthropynewsdigest.org/feeds/rfps.rss, overridable via PND_RFP_FEED_URL).

• Client — src/sources/pnd-rss/client.ts: fetchPndRfpFeed(feedUrl) does a plain fetch, throwing on any non-2xx response.
• Normalize — src/sources/pnd-rss/normalize.ts: parsePndFeed(xml) is a pure function (fast-xml-parser's XMLParser, with isArray forcing <item> to always parse as an array so single-item feeds don't collapse to a bare object) that maps each RSS <item> to a NewGrantInput:
  • title → title; link → sourceUrl (items with no link are skipped — there's nothing stable to upsert-key on)
  • description → synopsis, after stripping HTML tags and decoding named + numeric entities
  • funder — best-effort regex over the title, then the description: captures the leading proper-noun phrase before an announcing verb (accepting / invites / seeks / announces / offers); falls back to 'See RFP'
  • closeDate — best-effort regex for Deadline: <date> / due <date> phrasing (both month-name and ISO-ish date forms); invalid or absent → null
  • applicationEffortEstimate: 'unknown', source: 'pnd_rss', openDate: null, lastVerifiedAt: new Date()
• Workflow — src/workflows/ingest-pnd-rss.ts: scheduled nightly at 30 3 * * * (ingestPndRss), following the same registration pattern as ingest-grants.ts (module-scope deps registry populated via setIngestPndRssDeps before DBOS.launch(), dual workflow+scheduled registration, globalThis double-registration guard). Steps: fetch (3 retries) → parse (pure) → upsert via serverInsertGrants in batches of 100.

ProPublica Nonprofit Explorer enrichment

Daily scheduled workflow (enrichOrgs, cron 0 5 * * *) that fills in IRS-derived fields — EIN, NTEE code, most-recent-filing total revenue, fiscal year-end month — for orgs discovered by the other ingestion sources but never resolved against the IRS.

Flow:

1. serverListOrgsNeedingEnrichment(db, { limit: 200 }) — orgs with no EIN and no revenue on file, oldest-updatedAt first.
2. For each org, sequentially (never forEach+async — the client's politeness delay depends on awaiting each call before starting the next):
   • searchOrganizations(org.name, org.state) against Nonprofit Explorer's search.json.
   • pickBestMatch({ name, city }, candidates) — pure, conservative fuzzy match (see below). Returns null rather than guess when ambiguous.
   • On no match: serverEnrichOrg(db, org.id, { ein: null, nteeCode: null, totalRevenue: null, fiscalYearEnd: null }) — stamps updatedAt so the org drops out of the next run's serverListOrgsNeedingEnrichment result without pretending to have real data. icpBandForRevenue still runs inside serverEnrichOrg and correctly returns 'unknown' for a null revenue.
   • On match: getOrganization(ein) for filing history, extractEnrichment(detail), then serverEnrichOrg(db, org.id, enrichment).
3. Per-org failures are caught, logged, and counted — one bad org doesn't kill the batch. If failures exceed 20% of the attempted batch, the workflow rethrows (systemic-failure signal for DBOS retry/alerting) after logging attempted/resolved/unresolved/failed counts.

Matching (src/sources/propublica/match.ts): normalizeOrgName lowercases, strips punctuation (apostrophes drop silently, other punctuation becomes a separator), strips legal-suffix/article noise tokens (inc, corp, the, of, nh, the phrase new hampshire), and collapses whitespace. pickBestMatch prefers an exact normalized-name match, falls back to token-set Jaccard similarity ≥ 0.8, and disqualifies any candidate whose known city differs from the target's known city — on both paths, since same-legal-name-different-town is exactly the ambiguous case worth refusing rather than guessing. Ties break by city match, then shortest Levenshtein distance on the normalized name. A wrong EIN silently poisons downstream revenue/ICP-band data with no cheap way to detect it later, so every ambiguous case resolves to null (org stays in next run's backlog) instead of a best-effort guess.

Extraction (src/sources/propublica/extract.ts): extractEnrichment zero-pads the numeric EIN to 9 digits, passes through ntee_code (null-safe), and — from filings_with_data — picks the filing with the highest tax_prd_yr for totalRevenue and derives fiscalYearEnd as the zero-padded MM from that filing's tax_prd (YYYYMM, e.g. 202306 → '06'). Empty filing history yields totalRevenue: null, fiscalYearEnd: null.

Wiring: apps/outreach-worker/src/main.ts imports setEnrichOrgsDeps from ./workflows/enrich-orgs.js (which also registers the workflow/scheduled function as an import side effect) and calls setEnrichOrgsDeps({ db }) before DBOS.launch(), alongside the existing setIngestGrantsDeps/setExpireGrantsDeps calls.

NHDOJ Charitable Trusts registry

Monthly re-scan of the NH Department of Justice Charitable Trusts Unit's registry PDF — a roster of registered charitable organizations, not a grants feed. This is the org side of the pipeline: it upserts into orgs (keyed on case-insensitive (name, city, state), same as the ProPublica enrichment source), not grants. A brand-new registrant (inserted: true) is itself a signal worth tracking downstream — it's a segment often actively seeking first-time funding.

• Extract — src/sources/nhdoj/extract-pdf-text.ts: extractPositionedText(pdfBytes) is the only impure layer. Uses pdfjs-dist's legacy Node build (pdfjs-dist/legacy/build/pdf.mjs) with getDocument({ data, useSystemFonts: true }), returning one PositionedTextItem[] ({ str, x, y }, taken from each text item's transform matrix [4]/[5]) per page.
  • Node-runtime caveat: no disableWorker option exists on DocumentInitParameters, and none is needed — the legacy build self-detects isNodeJS at module load and unconditionally sets PDFWorker.#isWorkerDisabled = true, falling back to an in-process "fake worker" automatically (see PDFWorker#_initialize in the bundled pdf.mjs). No GlobalWorkerOptions.workerSrc setup is required for text extraction.
• Parse — src/sources/nhdoj/parse-registry.ts: pure row reconstruction over PositionedTextItem[][], entirely unit-testable on synthetic fixtures (no real PDF needed).
  • reconstructRegistryRows(pages, options?) groups items into visual lines by y (±2pt tolerance, configurable), buckets each line's items into 3 columns (name/city/status) by x-position, and stitches multi-line org names back together — a continuation line has text only in the name column. Column boundaries are inferred per page from that page's header row token x-positions (Organization/Name, City, Status); pages whose header doesn't repeat (e.g. page 2+) reuse the last-inferred boundaries, or accept an explicit columnBoundaries: [nameX, cityX, statusX] override. Throws if a page has data but no boundaries can be determined at all — a layout change should fail loudly, not misparse silently.
  • normalizeRegistryRows(rows) maps free-text status onto 'good_standing' (good standing / current / active) | 'lapsed' (lapsed / delinquent / suspended / expired / revoked) | 'unknown' (anything else, including blank), collapses whitespace in names/cities, and drops non-registrant artifact rows (repeated header, page-number footers, Page X of Y, NHDOJ letterhead).
• Workflow — src/workflows/ingest-nhdoj-orgs.ts: scheduled monthly at 0 4 1 * * (ingestNhdojOrgs), same registration pattern as ingest-grants.ts (module-scope deps registry via setIngestNhdojOrgsDeps, dual workflow+scheduled registration, globalThis guard). Steps: fetch PDF bytes from NHDOJ_REGISTRY_PDF_URL (3 retries) → extract + parse (pure, not a DBOS step) → serverUpsertOrgFromRegistry per row in a plain for...of loop (sourceRegistry: 'nhdoj_charitable_trusts') → console.log of rows parsed / upserted / newly-inserted counts.
  • Config gap ≠ outage: if NHDOJ_REGISTRY_PDF_URL is unset, the workflow logs a console.warn and returns without throwing. NHDOJ has no stable URL for the registry PDF — it changes whenever they republish — so a hard failure here would page on-call for a config gap rather than a real problem.
  • Not yet wired into apps/outreach-worker/src/main.ts — the module registers itself as a side effect of being imported (per the pattern above), but main.ts needs an explicit import (for registration-before-launch ordering) plus a setIngestNhdojOrgsDeps({ db }) call before DBOS.launch():

    import { setIngestNhdojOrgsDeps } from './workflows/ingest-nhdoj-orgs.js';
    // ...
    setIngestNhdojOrgsDeps({ db });
    

First-run field findings (2026-07-16)

• NHDOJ registry: 8-column layout (Reg. No. | Charity Name | Address | City | State | Zip | Status | Report Due), single-letter statuses (G/X/S legend), includes out-of-state charities registered to solicit in NH. Parsed 13,709 registrants → 13,632 orgs (6,266 NH). Reg. No. is the stable upsert key (orgs.registration_number). mm.nh.gov sits behind Akamai: bare curl gets 403; Node fetch with browser-like headers passes (client sends them). NHDOJ_REGISTRY_PDF_PATH overrides the URL for supervised runs.
• ProPublica: a state-scoped search with zero hits returns 404, not an empty list — the client maps 404 → no candidates (org resolves unresolved). Before the fix this tripped the 20% systemic-failure breaker.
• Grants.gov: first pass ingested 200 opportunities (search cap 1000 / detail cap 200); the hourly expiry sweep correctly expired a same-day deadline immediately.
• Enrichment queue prioritizes NH + good-standing orgs before the out-of-state tail; ~60% of NH orgs resolve an EIN per batch, the rest record an all-null attempt.