christian/grant-outreach-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ingestion): survive first contact with real data sources

Browse Source 
NHDOJ: parser rebuilt for the real 8-column registry layout (Reg. No. |
Charity Name | Address | City | State | Zip | Status | Report Due) with
single-letter G/X/S statuses; Reg. No. is the stable upsert key (new
orgs.registration_number column + partial unique index, enum gains
'suspended' via idempotent ADD VALUE); out-of-state registrants keep
their real state. Akamai-safe fetch headers + NHDOJ_REGISTRY_PDF_PATH
local-file override.

ProPublica: zero-hit state-scoped searches return 404, not an empty
list — map to no-candidates instead of failure (tripped the systemic-
failure breaker at 60/200 on first contact). Enrichment queue now
prioritizes NH good-standing orgs over the out-of-state tail.

PND: feed retired upstream (HTML shell on every historical path) —
documented as rework candidate, low priority.

run-once.ts: supervised one-off runner through the durable DBOS
handles (workflow modules now export run*Now accessors); drop the
double pool.end() after DBOS.shutdown().

First supervised run: 200 Grants.gov opportunities (1 auto-expired),
13,632 orgs from the 427-page registry, enrichment at failed=0 with
121/200 EIN resolution in the NH-priority batch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Croissant Le Doux
2026-07-16 16:20:47 -04:00
parent fe12b032af
commit 8d57d57557
17 changed files with 1898 additions and 288 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/features/ingestion.md
View File

@@ -47,6 +47,13 @@ Grant upserts key on `grants.source_url`; org registry upserts key on case-insen
### Philanthropy News Digest RSS
> **Status (2026-07-16): feed retired upstream.** Every historical feed path
> (`/feeds/rfps.rss` and variants) now returns the site's HTML shell — PND is
> a Next.js app with client-side data and no `<link rel=alternate>` feeds.
> The workflow runs and upserts zero rows. Rework candidate: drive their
> internal JSON API / `__NEXT_DATA__` instead. Low priority — PND was "cheap
> incremental coverage"; Grants.gov + NH state sources carry the corpus.
Nightly ingestion of the Philanthropy News Digest "RFPs" RSS feed (`https://philanthropynewsdigest.org/feeds/rfps.rss`, overridable via `PND_RFP_FEED_URL`).
- **Client** — `src/sources/pnd-rss/client.ts`: `fetchPndRfpFeed(feedUrl)` does a plain `fetch`, throwing on any non-2xx response.
@@ -94,3 +101,10 @@ Monthly re-scan of the NH Department of Justice Charitable Trusts Unit's registr
// ...
setIngestNhdojOrgsDeps({ db });
```
## First-run field findings (2026-07-16)
- **NHDOJ registry**: 8-column layout (`Reg. No. | Charity Name | Address | City | State | Zip | Status | Report Due`), single-letter statuses (G/X/S legend), includes out-of-state charities registered to solicit in NH. Parsed 13,709 registrants → 13,632 orgs (6,266 NH). `Reg. No.` is the stable upsert key (`orgs.registration_number`). mm.nh.gov sits behind Akamai: bare curl gets 403; Node fetch with browser-like headers passes (client sends them). `NHDOJ_REGISTRY_PDF_PATH` overrides the URL for supervised runs.
- **ProPublica**: a state-scoped search with zero hits returns **404**, not an empty list — the client maps 404 → no candidates (org resolves `unresolved`). Before the fix this tripped the 20% systemic-failure breaker.
- **Grants.gov**: first pass ingested 200 opportunities (search cap 1000 / detail cap 200); the hourly expiry sweep correctly expired a same-day deadline immediately.
- **Enrichment queue** prioritizes NH + good-standing orgs before the out-of-state tail; ~60% of NH orgs resolve an EIN per batch, the rest record an all-null attempt.