feat(ingestion): Phase 1 data spine — Grants.gov, PND RSS, NHDOJ registry, ProPublica enrichment
Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry (case-insensitive name/city/state key, new-registrant signal), serverEnrichOrg (IRS fields + re-band, all-null marks attempted), serverListOrgsNeedingEnrichment. Worker: four source verticals, each a thin fail-loud client + pure tested normalize layer + DBOS scheduled workflow: - ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity details (cap 200, logged drops, 250ms politeness) -> batched upsert - ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/ deadline extraction, link-keyed upsert - ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure row reconstruction (multi-line names, inferred columns, fail-loud on layout change), registry upsert; no-op warn when PDF URL unset - enrichOrgs (daily): ProPublica search -> conservative name/city match (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for checkpointed resume, >20% batch failure rethrows 94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -8,7 +8,8 @@
|
||||
"dev": "node --env-file=.env --env-file-if-exists=.env.local --import tsx/esm ./src/main.ts",
|
||||
"build": "esbuild src/main.ts --bundle --platform=node --format=esm --outfile=build/main.js --packages=external",
|
||||
"start": "node ./build/main.js",
|
||||
"typecheck": "tsc --noEmit"
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test": "vitest run"
|
||||
},
|
||||
"imports": {
|
||||
"#~/*": "./src/*"
|
||||
@@ -18,6 +19,8 @@
|
||||
"@dbos-inc/drizzle-datasource": "4.17.6",
|
||||
"@novelpad/outreach-core": "workspace:^",
|
||||
"drizzle-orm": "0.44.6",
|
||||
"fast-xml-parser": "^4.5.0",
|
||||
"pdfjs-dist": "^4.10.38",
|
||||
"pg": "8.20.0",
|
||||
"tsx": "^4.19.2"
|
||||
},
|
||||
@@ -26,6 +29,7 @@
|
||||
"@types/node": "^22",
|
||||
"@types/pg": "8.20.0",
|
||||
"esbuild": "^0.24.0",
|
||||
"typescript": "^5.9.3"
|
||||
"typescript": "^5.9.3",
|
||||
"vitest": "^3.2.4"
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user