1735ff6754
Core: icpBandForRevenue (100K-5M primary), serverUpsertOrgFromRegistry (case-insensitive name/city/state key, new-registrant signal), serverEnrichOrg (IRS fields + re-band, all-null marks attempted), serverListOrgsNeedingEnrichment. Worker: four source verticals, each a thin fail-loud client + pure tested normalize layer + DBOS scheduled workflow: - ingestGrants (nightly): Search2 paginated (cap 1000) -> fetchOpportunity details (cap 200, logged drops, 250ms politeness) -> batched upsert - ingestPndRss (nightly): RSS via fast-xml-parser, heuristic funder/ deadline extraction, link-keyed upsert - ingestNhdojOrgs (monthly): pdfjs-dist positioned-text extraction, pure row reconstruction (multi-line names, inferred columns, fail-loud on layout change), registry upsert; no-op warn when PDF URL unset - enrichOrgs (daily): ProPublica search -> conservative name/city match (null beats guess) -> latest-filing revenue/NTEE/FYE, per-org steps for checkpointed resume, >20% batch failure rethrows 94 worker + 16 core + 5 ai tests green; docs/features/ingestion.md added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
23 lines
1.2 KiB
Plaintext
23 lines
1.2 KiB
Plaintext
# Postgres connection string shared by the DBOS system schema, the pg Pool,
|
|
# and the Drizzle client over the outreach schema. Required — main.ts throws
|
|
# on boot if this is unset.
|
|
DATABASE_URL=postgres://postgres:postgres@localhost:5432/helmdocs_outreach
|
|
|
|
# Path to a GCP service account key JSON used by ingestion/scoring steps that
|
|
# call Google-hosted APIs (e.g. @google/genai subscoring, Drive-backed org
|
|
# profile lookups). Not read directly by main.ts yet; present here so it's
|
|
# provisioned alongside DATABASE_URL for the workflows that will need it.
|
|
GCP_SERVICE_ACCOUNT_KEY_PATH=./secrets/gcp-service-account.json
|
|
|
|
# Optional: cap the pg Pool size (defaults to 20 — see main.ts).
|
|
# PG_POOL_MAX=20
|
|
|
|
# NHDOJ Charitable Trusts registry PDF — the URL changes whenever NHDOJ
|
|
# republishes (roughly monthly). When unset, the monthly ingestNhdojOrgs
|
|
# workflow logs a warning and no-ops instead of failing.
|
|
# NHDOJ_REGISTRY_PDF_URL=https://www.doj.nh.gov/.../charitable-trusts-registry.pdf
|
|
|
|
# Optional override for the Philanthropy News Digest RFP feed (defaults to
|
|
# the public feed URL baked into src/sources/pnd-rss/client.ts).
|
|
# PND_RFP_FEED_URL=https://philanthropynewsdigest.org/feeds/rfps.rss
|