Draw the HelmDocs integration boundary #19

New Issue

christian · 2026-08-02T03:05:54Z

christian commented

2026-08-02 03:05:54 +00:00

Question

The existing bidder engine — drafting, compliance matrix, go/no-go, skeleton, export — already works and was proven on real solicitations. The grilling surfaced that this may be the actual moat rather than the later enhancement it was framed as, because native authoring is the only path that produces genuinely clean structured responses.

Resolve: the interface between the greenfield transmission system and HelmDocs (API shape, MCP, auth, identity mapping); what flows each way; whether v1 ships any integration or only an interface designed to accept one; and how a vendor authoring natively gets a better outcome without native authoring becoming mandatory.

Parent: #1

## Question The existing bidder engine — drafting, compliance matrix, go/no-go, skeleton, export — already works and was proven on real solicitations. The grilling surfaced that this may be the actual moat rather than the later enhancement it was framed as, because native authoring is the only path that produces genuinely clean structured responses. Resolve: the interface between the greenfield transmission system and HelmDocs (API shape, MCP, auth, identity mapping); what flows each way; whether v1 ships any integration or only an interface designed to accept one; and how a vendor authoring natively gets a better outcome without native authoring becoming mandatory. --- Parent: #1

christian added the

wayfinder:grilling

wayfinder:ticket

labels 2026-08-02 03:05:54 +00:00

christian added a new dependency 2026-08-02 03:06:26 +00:00

#10 Define the common schema for RFPs and responses

christian added a new dependency 2026-08-02 03:06:26 +00:00

#11 Define the tenancy, identity, and authorization model

christian commented

2026-08-03 22:19:00 +00:00

Reframe: extraction is the first thing to cross this boundary

This ticket was drawn abstractly. It now has a concrete first subject.

novelpad-desktop already contains a production extraction pipeline at
apps/website/app/rag/solicitation-preprocessor/ - roughly twenty files, each with tests:

a smart router (page-profile.ts, route-pages.ts) choosing an extractor per page
a free tier (pdfjs, buffer in, no cloud call and no cost)
Document AI Layout Parser with sync (30pp cap), chunked, and batch modes
sub-PDF splitting - 30pp chunks, 18 MB cap, max 4 subdivisions for dense pages
Form Parser, PaddleOCR, office/DOCX, and a hybrid merge that OCRs only the pages the
free tier failed, at confidence 0.5
text-validity.ts to detect bad text layers, per-extraction costUsd, and a versioned
self-healing reprocess policy that deliberately never re-runs ready rows because Document AI is
billable

Why this lands here. The map's standing preference is reuse the thinking and the prompts, not
the runtime. That was written before anyone had looked at the size of the runtime in question.
Re-implementing this for v1 would duplicate a smart router, a cost model and an OCR fallback ladder
for no gain - but importing it wholesale contradicts the separate-deployable decision from #2.
"Call it across an API boundary" is exactly what this ticket exists to decide, and extraction is
the first real thing to cross.

Three facts to decide against, all verified rather than assumed:

It is coupled to GCS at module scope. Importing free-tier-extract.server.ts throws on a
missing GCP_BUCKET_NAME, then on GCP_SERVICE_ACCOUNT_KEY_PATH - even for the buffer path
that makes no cloud call. Reuse means either supplying credentials or refactoring the storage
import out of module scope.
Layout Parser output is what #16 needs. It returns OutlineNode structure and per-page
sectionPath, which is the source for requirement.provenance_span and the input to span
coverage. That was the shakiest assumption in the completeness decision and it now has a real
backing.
Cost is per-extraction and real. Corpus iteration and any re-extraction have a bill attached,
which is an argument for the boundary being a service call with caching rather than a library
import.

Also relevant: export-fixture.server.ts already emits a committable fixture layout
(extraction.json + pages.json, embeddings excluded). #30's ground-truth files can start from
pipeline output and be hand-corrected, rather than authored blind.

## Reframe: extraction is the first thing to cross this boundary This ticket was drawn abstractly. It now has a concrete first subject. **`novelpad-desktop` already contains a production extraction pipeline** at `apps/website/app/rag/solicitation-preprocessor/` - roughly twenty files, each with tests: - a **smart router** (`page-profile.ts`, `route-pages.ts`) choosing an extractor per page - a **free tier** (pdfjs, buffer in, no cloud call and no cost) - **Document AI Layout Parser** with sync (30pp cap), chunked, and batch modes - **sub-PDF splitting** - 30pp chunks, 18 MB cap, max 4 subdivisions for dense pages - **Form Parser**, **PaddleOCR**, **office/DOCX**, and a hybrid merge that OCRs only the pages the free tier failed, at confidence 0.5 - `text-validity.ts` to detect bad text layers, per-extraction `costUsd`, and a versioned self-healing reprocess policy that deliberately never re-runs `ready` rows because Document AI is billable **Why this lands here.** The map's standing preference is *reuse the thinking and the prompts, not the runtime*. That was written before anyone had looked at the size of the runtime in question. Re-implementing this for v1 would duplicate a smart router, a cost model and an OCR fallback ladder for no gain - but importing it wholesale contradicts the separate-deployable decision from https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/2. **"Call it across an API boundary" is exactly what this ticket exists to decide**, and extraction is the first real thing to cross. **Three facts to decide against**, all verified rather than assumed: 1. **It is coupled to GCS at module scope.** Importing `free-tier-extract.server.ts` throws on a missing `GCP_BUCKET_NAME`, then on `GCP_SERVICE_ACCOUNT_KEY_PATH` - *even for the buffer path that makes no cloud call*. Reuse means either supplying credentials or refactoring the storage import out of module scope. 2. **Layout Parser output is what https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/16 needs.** It returns `OutlineNode` structure and per-page `sectionPath`, which is the source for `requirement.provenance_span` and the input to span coverage. That was the shakiest assumption in the completeness decision and it now has a real backing. 3. **Cost is per-extraction and real.** Corpus iteration and any re-extraction have a bill attached, which is an argument for the boundary being a *service call with caching* rather than a library import. Also relevant: `export-fixture.server.ts` already emits a committable fixture layout (`extraction.json` + `pages.json`, embeddings excluded). https://gitea.stephenmann.io/christian/helmdocs-proposal-system/issues/30's ground-truth files can start from pipeline output and be hand-corrected, rather than authored blind.

christian referenced this issue

2026-08-03 22:19:00 +00:00

Build the synthesized response corpus #30

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Depends on

#10 Define the common schema for RFPs and responses

christian/helmdocs-proposal-system

#11 Define the tenancy, identity, and authorization model

christian/helmdocs-proposal-system

Reference: christian/helmdocs-proposal-system#19