christian/grant-outreach-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ingestion): drain the Grants.gov backlog + source-filtered review queue

Browse Source 
ingest-grants spent its 200-detail budget on hits.slice(0, 200) — the
same head of the search results every night; the backlog never drained.
Now new opportunities fill the budget first (serverListGrantSourceUrls
partition), remaining budget refreshes known ones; search cap raised to
2,000. Full eligible pool turns out to be 565 federal opportunities —
drained in two passes, 366 newly embedded.

Review queue gains a source badge column and All/Foundations/Federal
RFPs filter (?source=) — foundation easy-wins otherwise bury posted-RFP
matches, which score lower by design (no precedent, national pools) but
are the deadline-driven sends. Immediate proof: DOJ OVW FY2026 DV
program (closes 9/8) matched five NH domestic-violence orgs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Croissant Le Doux
2026-07-16 22:27:41 -04:00
parent a92b75b315
commit 4fa0bb1c32
5 changed files with 83 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
apps/outreach-worker/src/workflows/ingest-grants.ts
View File

@@ -16,6 +16,7 @@ import { DBOS, SchedulerMode } from '@dbos-inc/dbos-sdk';
import type { schema } from '@novelpad/outreach-core';
import {
serverInsertGrants,
serverListGrantSourceUrls,
type NewGrantInput,
} from '@novelpad/outreach-core/server';
import type { NodePgDatabase } from 'drizzle-orm/node-postgres';
@@ -35,7 +36,7 @@ export type OutreachDb = NodePgDatabase<typeof schema>;
* endpoint per run. Search hits are cheap (one paginated request per ~100),
* so this is generous headroom above the actual nightly posting volume.
*/
const SEARCH_HIT_CAP = 1000;
const SEARCH_HIT_CAP = 2_000;
/**
* Cap on how many opportunities get a `fetchOpportunity` detail call per
* run. Detail fetches are one request per grant (plus a politeness delay),
@@ -92,14 +93,22 @@ interface HitWithDetail {
* this run.
*/
async function fetchGrantDetails(
db: OutreachDb,
hits: ReadonlyArray<GrantsGovSearchHit>,
): Promise<ReadonlyArray<HitWithDetail>> {
const toFetch = hits.slice(0, DETAIL_FETCH_CAP);
if (hits.length > toFetch.length) {
console.warn(
`[ingest-grants] detail-fetch cap ${DETAIL_FETCH_CAP} reached: dropping ${hits.length - toFetch.length} of ${hits.length} opportunities this run`,
);
}
// Spend the per-run detail budget on NEW opportunities first — the old
// hits.slice(0, cap) re-fetched the same head of the search results
// every night and never drained the backlog. Known opportunities fill
// any remaining budget (refreshing close dates / lastVerifiedAt).
const known = await serverListGrantSourceUrls(db, 'grants_gov');
const isKnown = (hit: GrantsGovSearchHit) =>
known.has(`https://www.grants.gov/search-results-detail/${hit.id}`);
const fresh = hits.filter((h) => !isKnown(h));
const refresh = hits.filter(isKnown);
const toFetch = [...fresh, ...refresh].slice(0, DETAIL_FETCH_CAP);
console.log(
`[ingest-grants] detail budget ${DETAIL_FETCH_CAP}: ${Math.min(fresh.length, DETAIL_FETCH_CAP)} new, ${Math.max(0, Math.min(DETAIL_FETCH_CAP - fresh.length, refresh.length))} refresh, backlog remaining ${Math.max(0, fresh.length - DETAIL_FETCH_CAP)}`,
);
const details = await fetchOpportunityDetails(toFetch.map((hit) => hit.id));
return toFetch.map((hit, i) => {
@@ -147,7 +156,7 @@ async function runIngestGrants(): Promise<void> {
const { db } = getIngestGrantsDeps();
const hits = await searchGrantsGovStep();
const pairs = await fetchGrantDetailsStep(hits);
const pairs = await fetchGrantDetailsStep(db, hits);
const normalized = normalize(pairs);
await upsertGrantsStep(db, normalized);
}