christian/grant-outreach-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(scoring): 990-PF funder-precedent index — the 25-point subscore goes live

Browse Source 
New funders/funder_grants schema + ingest990pf monthly workflow: IRS BMF
state file discovers NH private foundations (747), e-file index CSVs
select their latest 990-PF filings, batch ZIPs stream through fflate
(4/run cap, most-hits-first, deferred logged), grants-paid rows land in
funder_grants, and funders with >=2 NH grants synthesize rolling grant
rows (source irs_990pf, funder_ein linked) that flow through the
existing embed+match pipeline.

Scoring v2: funderPrecedentSubscore tiers repeated in-state giving
(1/3/5/10 -> 8/15/20/25); easy win = >=65 total AND >=12 precedent
(plan's precedent floor); scale is the full 0-100. Rolling deadlines
pass the runway gate. Retrieval computes per-funder in-state counts and
exposes funder_ein.

Lead-quality gates from the first precedent run's failures: candidate
orgs exclude NTEE T* grantmakers; self-matches gated by EIN + normalized
name (NHDOJ registers foundations as charities, several without resolved
EINs — the first run's top 'leads' were foundations matched to
themselves).

Live: ~6.5GB of IRS batches processed, 2,766 grants-paid rows, 123
synthesized foundation grants, 89 easy wins across 27 orgs, credible
top-10 (AIDS Response-Seacoast -> Foundation for Seacoast Health, 25/25
precedent). 153 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Croissant Le Doux
2026-07-16 18:17:28 -04:00
parent 0ee478ec3d
commit 63b58e514d
34 changed files with 3634 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/features/ingestion.md
View File

@@ -6,14 +6,15 @@ Grant upserts key on `grants.source_url`; org registry upserts key on case-insen
## Schedules
| Workflow | Cron (UTC) | Source |
|---|---|---|
| `ingestGrants` | `0 3 * * *` | Grants.gov Search2 |
| `ingestPndRss` | `30 3 * * *` | Philanthropy News Digest RFP feed |
| `expireGrants` | `0 * * * *` | (sweep: `status='expired'` past `close_date`) |
| `enrichOrgs` | `0 5 * * *` | ProPublica Nonprofit Explorer |
| `embedGrants` | `15 4 * * *` | gemini-embedding-001 over open-grant synopses → `grants.synopsis_embedding` (first paid AI call; ~pennies/batch) |
| `ingestNhdojOrgs` | `0 4 1 * *` | NHDOJ Charitable Trusts registry PDF |
| Workflow | Cron (UTC) | Source |
| ----------------- | ------------ | ---------------------------------------------------------------------------------------------------------------- |
| `ingestGrants` | `0 3 * * *` | Grants.gov Search2 |
| `ingestPndRss` | `30 3 * * *` | Philanthropy News Digest RFP feed |
| `expireGrants` | `0 * * * *` | (sweep: `status='expired'` past `close_date`) |
| `enrichOrgs` | `0 5 * * *` | ProPublica Nonprofit Explorer |
| `embedGrants` | `15 4 * * *` | gemini-embedding-001 over open-grant synopses → `grants.synopsis_embedding` (first paid AI call; ~pennies/batch) |
| `ingestNhdojOrgs` | `0 4 1 * *` | NHDOJ Charitable Trusts registry PDF |
| `ingest990pf` | `0 6 2 * *` | IRS BMF + e-file index + batch ZIPs → funders/funder_grants → synthesized foundation grants |
## Sources
@@ -26,16 +27,19 @@ Grant upserts key on `grants.source_url`; org registry upserts key on case-insen
**Cadence**: Nightly, `0 3 * * *` (03:00 UTC), registered as a DBOS scheduled workflow (`ingestGrants`, `ExactlyOncePerInterval`).
**API**: Public Grants.gov Search2 API, no key required.
- `POST https://api.grants.gov/v1/api/search2` — enumerates opportunities, filtered to `oppStatuses: 'posted'` and nonprofit eligibility codes `12|13` (501(c)(3) and non-501(c)(3) nonprofits). Paginated via `rows`/`startRecordNum`.
- `POST https://api.grants.gov/v1/api/fetchOpportunity` — full detail (`synopsis`: description, applicant types, award amounts, response date) for a single opportunity id.
- Both endpoints wrap responses in an `{ errorcode, msg, data }` envelope. Any non-2xx HTTP status or `errorcode !== 0` throws immediately with a descriptive message — the client never silently drops or swallows an upstream failure.
**Caps** (per nightly run):
- Search enumeration: up to **1000** hits (`SEARCH_HIT_CAP`). Count logged via `console.log`.
- Detail fetch: up to **200** opportunities (`DETAIL_FETCH_CAP`), each a separate `fetchOpportunity` call with a **~250ms** politeness delay between requests. When the search result set exceeds the cap, the excess is dropped for that run (picked up on a later run) and a `console.warn` states exactly how many opportunities were skipped — never a silent truncation.
- Upsert: batched in groups of **100** via `serverInsertGrants`.
**Normalization rules** (`normalizeGrantsGovOpportunity`, pure — no I/O):
- `sourceUrl` (the upsert key): `https://www.grants.gov/search-results-detail/{opportunityId}` — stable across re-crawls.
- `funder`: `agencyDetails.agencyName` → `synopsis.agencyName` → search hit's `agency` → an agency code fallback → literal `'Unknown federal agency'` if nothing is present.
- `synopsis`: HTML-stripped from `synopsis.synopsisDesc` (block tags become newlines, common entities decoded); `null` if empty after stripping.
@@ -71,6 +75,7 @@ Nightly ingestion of the Philanthropy News Digest "RFPs" RSS feed (`https://phil
Daily scheduled workflow (`enrichOrgs`, cron `0 5 * * *`) that fills in IRS-derived fields — EIN, NTEE code, most-recent-filing total revenue, fiscal year-end month — for orgs discovered by the other ingestion sources but never resolved against the IRS.
**Flow:**
1. `serverListOrgsNeedingEnrichment(db, { limit: 200 })` — orgs with no EIN and no revenue on file, oldest-`updatedAt` first.
2. For each org, sequentially (never `forEach`+async — the client's politeness delay depends on awaiting each call before starting the next):
- `searchOrganizations(org.name, org.state)` against Nonprofit Explorer's `search.json`.
@@ -79,7 +84,7 @@ Daily scheduled workflow (`enrichOrgs`, cron `0 5 * * *`) that fills in IRS-deri
- On match: `getOrganization(ein)` for filing history, `extractEnrichment(detail)`, then `serverEnrichOrg(db, org.id, enrichment)`.
3. Per-org failures are caught, logged, and counted — one bad org doesn't kill the batch. If failures exceed 20% of the attempted batch, the workflow rethrows (systemic-failure signal for DBOS retry/alerting) after logging attempted/resolved/unresolved/failed counts.
**Matching (`src/sources/propublica/match.ts`):** `normalizeOrgName` lowercases, strips punctuation (apostrophes drop silently, other punctuation becomes a separator), strips legal-suffix/article noise tokens (`inc`, `corp`, `the`, `of`, `nh`, the phrase `new hampshire`), and collapses whitespace. `pickBestMatch` prefers an exact normalized-name match, falls back to token-set Jaccard similarity ≥ 0.8, and disqualifies any candidate whose known city differs from the target's known city — on *both* paths, since same-legal-name-different-town is exactly the ambiguous case worth refusing rather than guessing. Ties break by city match, then shortest Levenshtein distance on the normalized name. A wrong EIN silently poisons downstream revenue/ICP-band data with no cheap way to detect it later, so every ambiguous case resolves to `null` (org stays in next run's backlog) instead of a best-effort guess.
**Matching (`src/sources/propublica/match.ts`):** `normalizeOrgName` lowercases, strips punctuation (apostrophes drop silently, other punctuation becomes a separator), strips legal-suffix/article noise tokens (`inc`, `corp`, `the`, `of`, `nh`, the phrase `new hampshire`), and collapses whitespace. `pickBestMatch` prefers an exact normalized-name match, falls back to token-set Jaccard similarity ≥ 0.8, and disqualifies any candidate whose known city differs from the target's known city — on _both_ paths, since same-legal-name-different-town is exactly the ambiguous case worth refusing rather than guessing. Ties break by city match, then shortest Levenshtein distance on the normalized name. A wrong EIN silently poisons downstream revenue/ICP-band data with no cheap way to detect it later, so every ambiguous case resolves to `null` (org stays in next run's backlog) instead of a best-effort guess.
**Extraction (`src/sources/propublica/extract.ts`):** `extractEnrichment` zero-pads the numeric EIN to 9 digits, passes through `ntee_code` (null-safe), and — from `filings_with_data` — picks the filing with the highest `tax_prd_yr` for `totalRevenue` and derives `fiscalYearEnd` as the zero-padded `MM` from that filing's `tax_prd` (`YYYYMM`, e.g. `202306` → `'06'`). Empty filing history yields `totalRevenue: null, fiscalYearEnd: null`.
@@ -98,11 +103,39 @@ Monthly re-scan of the NH Department of Justice Charitable Trusts Unit's registr
- **Config gap ≠ outage**: if `NHDOJ_REGISTRY_PDF_URL` is unset, the workflow logs a `console.warn` and returns without throwing. NHDOJ has no stable URL for the registry PDF — it changes whenever they republish — so a hard failure here would page on-call for a config gap rather than a real problem.
- **Not yet wired into `apps/outreach-worker/src/main.ts`** — the module registers itself as a side effect of being imported (per the pattern above), but `main.ts` needs an explicit import (for registration-before-launch ordering) plus a `setIngestNhdojOrgsDeps({ db })` call before `DBOS.launch()`:
```ts
import { setIngestNhdojOrgsDeps } from './workflows/ingest-nhdoj-orgs.js';
import { setIngestNhdojOrgsDeps } from "./workflows/ingest-nhdoj-orgs.js";
// ...
setIngestNhdojOrgsDeps({ db });
```
### IRS 990-PF funder precedent
Private foundations almost never post an open RFP — Grants.gov/PND/NHDOJ never see them. This vertical instead builds a "precedent index": it discovers NH-registered private foundations from the IRS Business Master File (BMF), cross-references the IRS e-file index to find each one's latest 990-PF filing, parses grants actually paid to NH nonprofits out of that filing's XML, and — for foundations with enough NH giving history — synthesizes a `grants` row summarizing that history as a lead ("this foundation funds orgs like you"). Unlike every other source in this doc, it upserts into **both** `funders`/`funder_grants` (the raw precedent data) and `grants` (the synthesized lead), keyed by `grants.source_url` like the rest.
**Source**: `apps/outreach-worker/src/sources/irs-990pf/` — `bmf.ts` (BMF fetch + parse), `index-csv.ts` (e-file index fetch + filter), `parse-990pf-xml.ts` (990-PF XML → grants paid + application info), `zip.ts` (batch-ZIP selective extraction). Wired into `apps/outreach-worker/src/workflows/ingest-990pf.ts`.
**Cadence**: Monthly, `0 6 2 * *` (06:00 UTC on the 2nd), registered as a DBOS scheduled workflow (`ingest990pf`, `ExactlyOncePerInterval`).
**Data sources**, all Akamai-fronted (same browser-header workaround as `fetchNhdojRegistryPdf`):
1. `https://www.irs.gov/pub/irs-soi/eo_nh.csv` — BMF state extract (~1.6MB). `PF_FILING_REQ_CD === '1'` identifies 990-PF filers.
2. `https://apps.irs.gov/pub/epostcard/990/xml/{year}/index_{year}.csv` — e-file index (~28MB). Filtered in a manual line-by-line scan (no full-file line array, no giant intermediate row array) to `RETURN_TYPE === '990PF'` rows whose EIN is in the BMF-discovered target set, keeping only the latest `TAX_PERIOD` per EIN.
3. `https://apps.irs.gov/pub/epostcard/990/xml/{year}/{XML_BATCH_ID}.zip` — batch ZIPs (100-400MB each), containing `{OBJECT_ID}_public.xml` per filing. Streamed straight to a temp file (`os.tmpdir()`, never buffered whole in the JS heap during download), then read back once and selectively extracted via `fflate`'s `unzipSync({ filter })` — only the wanted entries are inflated. The temp file is deleted after each batch (`finally`, so a mid-batch failure still cleans up).
**Caps** (per monthly run):
- Batches processed: up to **4** (`IRS_990PF_MAX_BATCHES_PER_RUN`), the batches with the most target-EIN hits first. Deferred batches are logged by id + hit count via `console.warn` — never a silent truncation. No `processed_batches` checkpoint is needed: re-parsing is idempotent (`serverReplaceFunderGrantsForYear` replaces per tax year) and the target object-id set shrinks on its own as funders' `latestTaxYear`/`latestObjectId` advance.
- Skip filter: an (EIN, filing) pair is skipped entirely — no download credited against it — when the e-file index's selected `OBJECT_ID` for that EIN already matches the funder's recorded `latestObjectId` (`serverListFunderLatestObjectIds`, `packages/outreach-core/src/funders/queries/list-funder-latest-object-ids.server.ts`), i.e. nothing changed since the last run.
- Years scanned: current + previous (`IRS_990PF_YEARS`, comma-sep override) — a filing's e-file index year need not equal its `TAX_PERIOD` year, so each selected filing tracks the actual index year its batch ZIP lives under (`IndexedFiling.indexYear` in the workflow), not a value re-derived from `TAX_PERIOD`.
**XML parsing** (`parse990PfXml`, pure — no I/O): `removeNSPrefix: true` handles filing-software-dependent namespace prefixes. Grants paid (`SupplementaryInformationGrp/GrantOrContributionPdDurYrGrp[]`) resolve the recipient name from `RecipientBusinessName/BusinessNameLine1Txt`, falling back to `RecipientPersonNm`; amounts round to whole dollars; a row with neither name field is dropped (no usable recipient identity). Application info (`ApplicationSubmissionInfoGrp`) captures whatever of `RecipientNm`/`FormAndInfoAndMaterialsTxt`/`SubmissionDeadlinesTxt`/`RestrictionsOnAwardsTxt` is present into a plain object, `null` if none are. Tax year prefers `ReturnHeader/TaxYr`, falling back to the calendar year of `TaxPeriodEndDt`, falling back (in the workflow, not the pure parser) to the filing's `TAX_PERIOD` (`YYYYMM`) when the XML has neither. Every group defaults to empty/`null` on absence rather than throwing — a missing Part XV section is normal for a foundation with no formal application process.
**Synthesis** (`buildSynthesizedGrant`, pure): for every funder with ≥2 grants paid into NH (`serverListFunderSynthesisData`), builds one `grants` row: `source: 'irs_990pf'`, `funderEin` set, `sourceUrl` the funder's ProPublica Nonprofit Explorer page (stable, human-followable, and distinct from the funder's own `funders.ein` upsert key so re-running never collides with a same-funder public-RFP row). `synopsis` is composed prose (location, in-state grant count + typical/median amount, up to 8 recent grant purposes, application-info sentence when present). `awardCeiling`/`awardFloor` derive from the aggregated median/max in-state amounts. `geographicScope` is deliberately left `null` — the synthesis query only sees grants paid _into_ NH, not the funder's total giving footprint, so there's no way to tell from this data whether NH is an exclusive restriction; asserting it would fake a hard geography gate the data doesn't support. `closeDate: null` (rolling), `applicationEffortEstimate: 'unknown'`, `status: 'open'`.
**Workflow** (`ingest-990pf.ts`): same registration pattern as `ingest-grants.ts` (module-scope deps registry via `setIngest990pfDeps`, dual workflow+scheduled registration, `globalThis` guard, `runIngest990PfNow()` accessor for `run-once.ts`). Steps: fetch BMF (3 retries) → upsert every parsed foundation as a `funders` row (per-row step, same pattern as `ingest-nhdoj-orgs.ts`) → fetch each target year's e-file index (3 retries, per-year) → pure `selectFilings` → skip-filter against recorded `latestObjectId` → group into batches, sort by hit count, cap → per batch (2 retries — a 100-400MB download is expensive to retry 3x): download, extract, parse, `serverUpsertFunder` (filing-derived fields) + `serverReplaceFunderGrantsForYear` per filing → synthesis step. Every stage logs its counts via `console.log`/`console.warn`.
**Core addition**: `packages/outreach-core/src/funders/queries/list-funder-latest-object-ids.server.ts` (`serverListFunderLatestObjectIds`) — the one core addition this vertical needed, following the existing action/query barrel pattern; everything else in `packages/outreach-core/src/funders/` (the `funders`/`funder_grants` schema, `serverUpsertFunder`, `serverReplaceFunderGrantsForYear`, `serverListFunderSynthesisData`) pre-existed this vertical.
## First-run field findings (2026-07-16)
- **NHDOJ registry**: 8-column layout (`Reg. No. | Charity Name | Address | City | State | Zip | Status | Report Due`), single-letter statuses (G/X/S legend), includes out-of-state charities registered to solicit in NH. Parsed 13,709 registrants → 13,632 orgs (6,266 NH). `Reg. No.` is the stable upsert key (`orgs.registration_number`). mm.nh.gov sits behind Akamai: bare curl gets 403; Node fetch with browser-like headers passes (client sends them). `NHDOJ_REGISTRY_PDF_PATH` overrides the URL for supervised runs.