Croissant Le Doux
|
1735ff6754
|
feat(ingestion): Phase 1 data spine — Grants.gov, PND RSS, NHDOJ registry, ProPublica enrichment
Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry
(case-insensitive name/city/state key, new-registrant signal),
serverEnrichOrg (IRS fields + re-band, all-null marks attempted),
serverListOrgsNeedingEnrichment.
Worker: four source verticals, each a thin fail-loud client + pure
tested normalize layer + DBOS scheduled workflow:
- ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity
details (cap 200, logged drops, 250ms politeness) -> batched upsert
- ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/
deadline extraction, link-keyed upsert
- ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure
row reconstruction (multi-line names, inferred columns, fail-loud on
layout change), registry upsert; no-op warn when PDF URL unset
- enrichOrgs (daily): ProPublica search -> conservative name/city match
(null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for
checkpointed resume, >20% batch failure rethrows
94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
2026-07-16 13:11:50 -04:00 |
|
Croissant Le Doux
|
14200edb60
|
feat: scaffold outreach engine monorepo on the novelpad-desktop stack
Workspaces: config (copied), outreach-core (schema + actions/queries +
hard gates), outreach-ai (Gemini client + embeddings copies, profiler and
mission-fit-judge agent stubs), outreach-worker (DBOS executor with
nightly ingest + hourly expiry workflows), outreach-review (RR7 review
queue v0). Initial drizzle migration incl. pgvector extension.
Stack contract: Yarn 4.5.0 + Turbo, Node 22.16, Drizzle 0.44.6 +
pgvector, DBOS 4.17.6, @google/genai on Vertex, gemini-embedding-001
@1536, React Router v7. Files copied from novelpad-desktop carry
provenance headers @ 62c56b87.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
2026-07-16 11:08:24 -04:00 |
|