grant-outreach-engine/docs/features/ingestion.md

# Data-spine ingestion (Phase 1)

How grant opportunities and NH nonprofit orgs enter the outreach database. All ingestion runs as DBOS scheduled workflows in `apps/outreach-worker`; every source has a thin impure client, a pure normalize layer (unit-tested on fixtures), and a workflow that upserts through `@novelpad/outreach-core/server` actions — never inline DB calls.

Grant upserts key on `grants.source_url`; org registry upserts key on case-insensitive `(name, city, state)`; IRS enrichment applies to org rows by id and re-bands ICP via `icpBandForRevenue` ($100K–$5M = primary).

## Schedules

| Workflow | Cron (UTC) | Source |
|---|---|---|
| `ingestGrants` | `0 3 * * *` | Grants.gov Search2 |
| `ingestPndRss` | `30 3 * * *` | Philanthropy News Digest RFP feed |
| `expireGrants` | `0 * * * *` | (sweep: `status='expired'` past `close_date`) |
| `enrichOrgs` | `0 5 * * *` | ProPublica Nonprofit Explorer |
| `ingestNhdojOrgs` | `0 4 1 * *` | NHDOJ Charitable Trusts registry PDF |

## Sources

<!-- Per-source sections assembled below -->

### Grants.gov

**Source**: `apps/outreach-worker/src/sources/grants-gov/` — `client.ts` (fetch), `normalize.ts` (pure transform), `pagination.ts` (pure pagination arithmetic). Wired into `apps/outreach-worker/src/workflows/ingest-grants.ts`.

**Cadence**: Nightly, `0 3 * * *` (03:00 UTC), registered as a DBOS scheduled workflow (`ingestGrants`, `ExactlyOncePerInterval`).

**API**: Public Grants.gov Search2 API, no key required.
- `POST https://api.grants.gov/v1/api/search2` — enumerates opportunities, filtered to `oppStatuses: 'posted'` and nonprofit eligibility codes `12|13` (501(c)(3) and non-501(c)(3) nonprofits). Paginated via `rows`/`startRecordNum`.
- `POST https://api.grants.gov/v1/api/fetchOpportunity` — full detail (`synopsis`: description, applicant types, award amounts, response date) for a single opportunity id.
- Both endpoints wrap responses in an `{ errorcode, msg, data }` envelope. Any non-2xx HTTP status or `errorcode !== 0` throws immediately with a descriptive message — the client never silently drops or swallows an upstream failure.

**Caps** (per nightly run):
- Search enumeration: up to **1000** hits (`SEARCH_HIT_CAP`). Count logged via `console.log`.
- Detail fetch: up to **200** opportunities (`DETAIL_FETCH_CAP`), each a separate `fetchOpportunity` call with a **~250ms** politeness delay between requests. When the search result set exceeds the cap, the excess is dropped for that run (picked up on a later run) and a `console.warn` states exactly how many opportunities were skipped — never a silent truncation.
- Upsert: batched in groups of **100** via `serverInsertGrants`.

**Normalization rules** (`normalizeGrantsGovOpportunity`, pure — no I/O):
- `sourceUrl` (the upsert key): `https://www.grants.gov/search-results-detail/{opportunityId}` — stable across re-crawls.
- `funder`: `agencyDetails.agencyName` → `synopsis.agencyName` → search hit's `agency` → an agency code fallback → literal `'Unknown federal agency'` if nothing is present.
- `synopsis`: HTML-stripped from `synopsis.synopsisDesc` (block tags become newlines, common entities decoded); `null` if empty after stripping.
- `awardFloor` / `awardCeiling` / `expectedAwardsCount`: parsed from string-or-number wire values (handles `"$1,500,000"`-style formatting), rounded to whole dollars/counts; `null` on unparseable input.
- `openDate` / `closeDate`: parsed from the search hit's `MM/DD/YYYY` fields (falling back to the detail's `postingDate`/`responseDate`); any invalid, out-of-range, or calendar-rollover date (e.g. `02/30/2026`) resolves to `null` rather than a garbage `Date`.
- `eligibilityEntityTypes`: trimmed, non-empty descriptions from `synopsis.applicantTypes`; `null` if none.
- `geographicScope`: always `null` — federal grants are national by default and Grants.gov has no reliable field to derive a narrower scope from; left unset rather than guessed.
- `programAreas`: `null` (not yet derived from this source).
- `applicationEffortEstimate`: `'unknown'`; `matchRequirement` / `applicationFormSupported`: `false`; `status`: `'open'`; `source`: `'grants_gov'`; `lastVerifiedAt`: set to the normalization timestamp.

### Philanthropy News Digest RSS

Nightly ingestion of the Philanthropy News Digest "RFPs" RSS feed (`https://philanthropynewsdigest.org/feeds/rfps.rss`, overridable via `PND_RFP_FEED_URL`).

- **Client** — `src/sources/pnd-rss/client.ts`: `fetchPndRfpFeed(feedUrl)` does a plain `fetch`, throwing on any non-2xx response.
- **Normalize** — `src/sources/pnd-rss/normalize.ts`: `parsePndFeed(xml)` is a pure function (`fast-xml-parser`'s `XMLParser`, with `isArray` forcing `<item>` to always parse as an array so single-item feeds don't collapse to a bare object) that maps each RSS `<item>` to a `NewGrantInput`:
  - `title` → `title`; `link` → `sourceUrl` (items with no link are skipped — there's nothing stable to upsert-key on)
  - `description` → `synopsis`, after stripping HTML tags and decoding named + numeric entities
  - `funder` — best-effort regex over the title, then the description: captures the leading proper-noun phrase before an announcing verb (`accepting` / `invites` / `seeks` / `announces` / `offers`); falls back to `'See RFP'`
  - `closeDate` — best-effort regex for `Deadline: <date>` / `due <date>` phrasing (both month-name and ISO-ish date forms); invalid or absent → `null`
  - `applicationEffortEstimate: 'unknown'`, `source: 'pnd_rss'`, `openDate: null`, `lastVerifiedAt: new Date()`
- **Workflow** — `src/workflows/ingest-pnd-rss.ts`: scheduled nightly at `30 3 * * *` (`ingestPndRss`), following the same registration pattern as `ingest-grants.ts` (module-scope deps registry populated via `setIngestPndRssDeps` before `DBOS.launch()`, dual workflow+scheduled registration, `globalThis` double-registration guard). Steps: fetch (3 retries) → parse (pure) → upsert via `serverInsertGrants` in batches of 100.

### ProPublica Nonprofit Explorer enrichment

Daily scheduled workflow (`enrichOrgs`, cron `0 5 * * *`) that fills in IRS-derived fields — EIN, NTEE code, most-recent-filing total revenue, fiscal year-end month — for orgs discovered by the other ingestion sources but never resolved against the IRS.

**Flow:**
1. `serverListOrgsNeedingEnrichment(db, { limit: 200 })` — orgs with no EIN and no revenue on file, oldest-`updatedAt` first.
2. For each org, sequentially (never `forEach`+async — the client's politeness delay depends on awaiting each call before starting the next):
   - `searchOrganizations(org.name, org.state)` against Nonprofit Explorer's `search.json`.
   - `pickBestMatch({ name, city }, candidates)` — pure, conservative fuzzy match (see below). Returns `null` rather than guess when ambiguous.
   - On no match: `serverEnrichOrg(db, org.id, { ein: null, nteeCode: null, totalRevenue: null, fiscalYearEnd: null })` — stamps `updatedAt` so the org drops out of the next run's `serverListOrgsNeedingEnrichment` result without pretending to have real data. `icpBandForRevenue` still runs inside `serverEnrichOrg` and correctly returns `'unknown'` for a null revenue.
   - On match: `getOrganization(ein)` for filing history, `extractEnrichment(detail)`, then `serverEnrichOrg(db, org.id, enrichment)`.
3. Per-org failures are caught, logged, and counted — one bad org doesn't kill the batch. If failures exceed 20% of the attempted batch, the workflow rethrows (systemic-failure signal for DBOS retry/alerting) after logging attempted/resolved/unresolved/failed counts.

**Matching (`src/sources/propublica/match.ts`):** `normalizeOrgName` lowercases, strips punctuation (apostrophes drop silently, other punctuation becomes a separator), strips legal-suffix/article noise tokens (`inc`, `corp`, `the`, `of`, `nh`, the phrase `new hampshire`), and collapses whitespace. `pickBestMatch` prefers an exact normalized-name match, falls back to token-set Jaccard similarity ≥ 0.8, and disqualifies any candidate whose known city differs from the target's known city — on *both* paths, since same-legal-name-different-town is exactly the ambiguous case worth refusing rather than guessing. Ties break by city match, then shortest Levenshtein distance on the normalized name. A wrong EIN silently poisons downstream revenue/ICP-band data with no cheap way to detect it later, so every ambiguous case resolves to `null` (org stays in next run's backlog) instead of a best-effort guess.

**Extraction (`src/sources/propublica/extract.ts`):** `extractEnrichment` zero-pads the numeric EIN to 9 digits, passes through `ntee_code` (null-safe), and — from `filings_with_data` — picks the filing with the highest `tax_prd_yr` for `totalRevenue` and derives `fiscalYearEnd` as the zero-padded `MM` from that filing's `tax_prd` (`YYYYMM`, e.g. `202306` → `'06'`). Empty filing history yields `totalRevenue: null, fiscalYearEnd: null`.

**Wiring:** `apps/outreach-worker/src/main.ts` imports `setEnrichOrgsDeps` from `./workflows/enrich-orgs.js` (which also registers the workflow/scheduled function as an import side effect) and calls `setEnrichOrgsDeps({ db })` before `DBOS.launch()`, alongside the existing `setIngestGrantsDeps`/`setExpireGrantsDeps` calls.

### NHDOJ Charitable Trusts registry

Monthly re-scan of the NH Department of Justice Charitable Trusts Unit's registry PDF — a roster of registered charitable organizations, not a grants feed. This is the org side of the pipeline: it upserts into `orgs` (keyed on case-insensitive `(name, city, state)`, same as the ProPublica enrichment source), not `grants`. A brand-new registrant (`inserted: true`) is itself a signal worth tracking downstream — it's a segment often actively seeking first-time funding.

- **Extract** — `src/sources/nhdoj/extract-pdf-text.ts`: `extractPositionedText(pdfBytes)` is the only impure layer. Uses `pdfjs-dist`'s legacy Node build (`pdfjs-dist/legacy/build/pdf.mjs`) with `getDocument({ data, useSystemFonts: true })`, returning one `PositionedTextItem[]` (`{ str, x, y }`, taken from each text item's transform matrix `[4]`/`[5]`) per page.
  - **Node-runtime caveat**: no `disableWorker` option exists on `DocumentInitParameters`, and none is needed — the legacy build self-detects `isNodeJS` at module load and unconditionally sets `PDFWorker.#isWorkerDisabled = true`, falling back to an in-process "fake worker" automatically (see `PDFWorker#_initialize` in the bundled `pdf.mjs`). No `GlobalWorkerOptions.workerSrc` setup is required for text extraction.
- **Parse** — `src/sources/nhdoj/parse-registry.ts`: pure row reconstruction over `PositionedTextItem[][]`, entirely unit-testable on synthetic fixtures (no real PDF needed).
  - `reconstructRegistryRows(pages, options?)` groups items into visual lines by y (±2pt tolerance, configurable), buckets each line's items into 3 columns (name/city/status) by x-position, and stitches multi-line org names back together — a continuation line has text only in the name column. Column boundaries are inferred per page from that page's header row token x-positions (`Organization`/`Name`, `City`, `Status`); pages whose header doesn't repeat (e.g. page 2+) reuse the last-inferred boundaries, or accept an explicit `columnBoundaries: [nameX, cityX, statusX]` override. Throws if a page has data but no boundaries can be determined at all — a layout change should fail loudly, not misparse silently.
  - `normalizeRegistryRows(rows)` maps free-text status onto `'good_standing'` (`good standing` / `current` / `active`) | `'lapsed'` (`lapsed` / `delinquent` / `suspended` / `expired` / `revoked`) | `'unknown'` (anything else, including blank), collapses whitespace in names/cities, and drops non-registrant artifact rows (repeated header, page-number footers, `Page X of Y`, NHDOJ letterhead).
- **Workflow** — `src/workflows/ingest-nhdoj-orgs.ts`: scheduled monthly at `0 4 1 * *` (`ingestNhdojOrgs`), same registration pattern as `ingest-grants.ts` (module-scope deps registry via `setIngestNhdojOrgsDeps`, dual workflow+scheduled registration, `globalThis` guard). Steps: fetch PDF bytes from `NHDOJ_REGISTRY_PDF_URL` (3 retries) → extract + parse (pure, not a DBOS step) → `serverUpsertOrgFromRegistry` per row in a plain `for...of` loop (`sourceRegistry: 'nhdoj_charitable_trusts'`) → `console.log` of rows parsed / upserted / newly-inserted counts.
  - **Config gap ≠ outage**: if `NHDOJ_REGISTRY_PDF_URL` is unset, the workflow logs a `console.warn` and returns without throwing. NHDOJ has no stable URL for the registry PDF — it changes whenever they republish — so a hard failure here would page on-call for a config gap rather than a real problem.
  - **Not yet wired into `apps/outreach-worker/src/main.ts`** — the module registers itself as a side effect of being imported (per the pattern above), but `main.ts` needs an explicit import (for registration-before-launch ordering) plus a `setIngestNhdojOrgsDeps({ db })` call before `DBOS.launch()`:
    ```ts
    import { setIngestNhdojOrgsDeps } from './workflows/ingest-nhdoj-orgs.js';
    // ...
    setIngestNhdojOrgsDeps({ db });
    ```