How Document Ingestion Works

The reserve-study pipeline: upload, extraction, ops review, reconciliation, release, and the customer archive decision.

Status: built and merged, not activated in production verified 2026-09-07

The application and engine code are merged and the v2 API surface is deployed, but production is not presenting active customer intake: the emergency pause flag is on, zero documents exist, and no property has ever recorded a v2 cutover. What activation still requires is in the activation runbook. Everything below describes the system as built and specified.

What it does

An authorized user selects a property, uploads its latest reserve-study PDF, and can leave the page. Gallopify durably tracks the document, sends it to the extraction service, gives an internal operator a real-source review and reconciliation workflow, applies the approved study to the property's planner, and asks the customer to decide whether projects absent from the new study should be archived.

V1 extracts projected expenditures only. The extraction engine is a black box whose quality can improve separately; v1 does not import funding schedules, balances, revenues, bank transactions, or compliance fields.

The stored state machine; customer labels beneath each mono state name. Transitions are validated with row locking in one service transaction, and the step out of in_review is the one atomic, idempotent release. returned, complete, abandoned, cancelled never reopen.

Product invariants

A document never disappears. Once its row exists, every transition is durable and every non-terminal document is recoverable after a crash or deploy.
No unreviewed extraction changes portal data. Ops approval is mandatory in v1 — no confidence-based auto-release.
Project identity survives. A matched project keeps its id; linked transactions remain linked.
The approved study is authoritative. Its values replace matched projects' values even when a customer edited the project since the previous study; the conflict is shown to ops and retained in the audit record.
Absence is not deletion. Projects absent from the new study remain active until the customer submits the archive decision; archive is soft and reversible.
Release is atomic and idempotent. A retry cannot duplicate projects or partially apply a study.
Approval cannot go stale silently. It binds to the exact extraction revision, reconciliation proposal, and project versions ops reviewed.
Environment boundaries are physical. Dev and production use different databases, source buckets, extraction deployments, and credentials.
The source shown to ops is real. The actual PDF sits beside the editable result; no fabricated citation highlights (the v1 engine provides no trustworthy ones).
Status is honest. Internal stack traces and engine details never appear in customer copy; authorized intake is available by default, never represented by shipping a disabled uploader.

The state machine

The stored states are deliberately more precise than the customer labels:

Stored status	Customer label	Terminal	Meaning
`awaiting_upload`	Uploading	no	Row exists; no object version registered. Older than 24 h → `abandoned`.
`received`	Queued for processing	no	Object verified, pinned, durably queued.
`submitting` / `extracting`	Processing	no	Worker lease owns engine submission; engine job polled.
`in_review`	In review	no	Immutable result exists; ops action required.
`awaiting_archive_decision`	Projects updated — review archives	no	Non-destructive changes live; missing projects still active.
`complete`	Complete	yes	Release applied, no archive action remains.
`returned`	Needs a new document	yes	Ops cannot validate it; customer-facing reason required.
`failed_final`	Processing failed	yes*	Retries exhausted; safe customer-facing reason. Admin retry possible; never blocks a replacement upload.
`abandoned` / `cancelled`	hidden / Property no longer active	yes	Never-completed upload cleaned up; or property archived before release.

Reconciliation: six exhaustive outcomes

Approved extracted rows are compared first with all active projects for the property (manually created or study-imported alike), then leftovers against archived projects so a reappearing component restores its identity instead of duplicating. Each row and each project participates in at most one match:

Outcome	Release behavior
`matched_unchanged`	Retain row and id; update source provenance.
`matched_changed`	Retain id; take the study's name, category, dates, amount. Flagged "changed" until acknowledged.
`matched_archived`	Restore an archived project's identity with incoming values; requires explicit ops confirmation.
`new`	Insert a new project; flagged "new" until acknowledged.
`archive_candidate`	Keep active; create the customer's archive/keep decision item.
`ambiguous`	Blocks release; ops must choose a match or explicitly mark the row new. Splits and merges are always ambiguous — v1 never attaches one study row to multiple projects.

The matching algorithm

Matching is deliberately conservative — it proposes work for ops, it does not create identity from a weak guess. False negatives create ops work; false positives corrupt identity.

Normalize names and categories (Unicode NFKC, lowercase, "&" → "and", punctuation removed, whitespace collapsed; no stemming, numbers kept).
Exact lock: a unique pair matching on normalized name, canonical category, and displayed start fiscal year is locked.
Weighted fuzzy score for remaining active pairs: 0.60 × name token-set similarity (0–100, one pinned implementation — the thresholds depend on it) + category (15 if identical canonical) + year (15 same / 10 within 1 / 5 within 2) + amount (10 within 5% / 7 within 15% / 3 within 30%), capped at 100.
Auto-propose only when the pair is each side's mutual best, scores ≥ 85, and each side's margin over its runner-up is ≥ 10.
Archived candidates are scored the same way afterward but always require explicit operator acceptance.
Any remaining active pair scoring ≥ 60 makes its connected component ambiguous (release-blocking); rows with no such evidence are new, projects not held by an ambiguous component are archive_candidate.

Thresholds are implementation constants gated by a human-labelled matching corpus: 100% precision required for automatic proposals; recall reported but not a ship blocker.

Manual edits: three-way comparison

For a project previously touched by a release, the latest release snapshot provides the base: if the current portal value differs from that base, the field is a manual-edit conflict — ops sees base, current, and incoming, and must acknowledge before approval applies the incoming value. Projects with no prior snapshot get an ordinary two-way diff. The release ledger, not a general versioning subsystem, supplies the history.

Ownership boundary

Gallopify owns the source-object key and document context, the state machine, immutable extraction revisions, ops claims/reviews/approvals, reconciliation proposals, releases, archive decisions, project rows, and notifications. The extraction engine owns only transient compute and its disposable artifacts — it never writes portal data and does not own review or release state. Gallopify integrates solely through an idempotent submit/status facade; the engine's other endpoints are internals, not a contract.

Object custody (S3 invariants)

Per-environment, versioning-enabled private buckets with per-environment credentials; no shared default bucket. Application principal, extraction principal, and bucket must belong to the same environment, and isolation probes must prove dev credentials cannot read production objects.
Keys from durable ids only (documents/{property}/{document}/source.pdf); user filenames are display metadata and never appear in a key.
Bytes never transit the backend: the backend creates the document row, then issues a 15-minute presigned POST constrained to the exact key, PDF content type, and a 1 byte–100 MB size range; the browser uploads directly to S3.
Exact-version pinning: completion supplies the object version id; the backend verifies metadata and magic bytes, stores the source hash, and pins the version. Extraction, review, download, and release always use the pinned version — a later write to the same key cannot change reviewed input.
Scoped access: Gallopify holds only presign-create, prefix-limited version enumeration, exact-version get/head, and exact-version delete; the engine holds read-only exact-version get for its environment prefix. Customer/ops downloads are authorized immediately before a five-minute exact-version presigned GET.
Retention: source PDFs live for the active-property lifetime plus 90 days after archival or account deletion; immutable result/review/release/audit history follows the financial-record retention policy (at least seven years). Storage operations fail closed with typed errors.

Availability and the emergency pause

A capabilities endpoint is the sole frontend availability source: every authorized active property reports enabled — there is no allowlist, canary cohort, or rollout mode. The only override is a global emergency pause flag (UPLOAD_PIPELINE_V2_UPLOADS_PAUSED): pause rejects new presigns while history, in-flight processing, review, archive, restore, and cleanup continue. It is an incident brake, not a rollout mechanism — and it is the flag currently holding production intake closed (see the runbook).

Eligible historical uploads from the retired legacy pipeline are imported metadata-only as read-only "Earlier uploads" — property-unassigned, visible only to their original uploader, never able to poll, retry, release, or block a v2 upload. The importer never copies source objects or invents provenance.

Authority and other locked decisions worth knowing

Any current canSee user may upload for a property — including assigned property and portfolio managers; upload does not require org-admin status. Property is selected and authorized at upload; release never infers it from the organization.
Archive/restore authority starts with the uploader and uploading-organization admins, falling back to current HOA admins, then CMC admins.
The v1 engine supplies no quarter: an extracted year becomes a full-fiscal-year project under the property's fiscal convention — quarter-level precision is never invented.
Ordinary project deletion is blocked only while that project has a pending archive decision.
Bank and budget ingestion remain unavailable until separately designed; /uploads is reserve-study-only.