RFP Ingest Pipeline

01

Discover

Sweep the pursuit folder and classify every file. Project manuals are the ingest target; everything else is noise or handled elsewhere.

idle

02

Crack

Each manual is converted to layout-aware text once and cached by content hash. This is the expensive step, so it never runs twice on the same bytes.

waiting

03

Index sections

Resolve every span of text to the section that owns it. Section numbers appear many times per volume — in the contents, the running footer, cross-references — so the index records boundaries, not mentions.

waiting

Manual	Div	Section	Title	Pages	Chunks

Every division is indexed, not just electrical. Filtering happens at query time so cross-trade questions stay answerable.

04

Embed & score

Chunks are embedded, then scored against the corpus by how often their language recurs. Standard specification text scores low; project-specific edits score high.

waiting

Boilerplate — recurs across the corpus Project-specific — rare, high signal

———

Nothing is discarded. The score rides alongside the vector as metadata, so similarity can down-weight boilerplate while extraction still reads it — and re-scoring as the corpus grows never forces a re-embed.

05

Extract

The structured branch reads the same cracked text and writes normalized fields, each carrying a confidence score so review can focus where the engine was unsure.

waiting