The same brief appears several times in search while an older, deleted or newly restricted copy can influence an answer. Nobody can see where the update stopped.
A reliable RAG ingestion pipeline uses a controlled lifecycle: discover, identify, fetch, parse, chunk, upsert, reconcile and expose state. Keep its four identities linked but separate. A webhook starts work; it does not prove current search.
This article owns ingestion and freshness. Source authority (part 3 of this guide), solution selection (part 4) and full evaluation and value (part 9) remain separate. No live tenant, corpus, parser, workflow, permission model, vector index or client outcome was tested.
The answer in 60 seconds
RAG ingestion pipeline:
- Discover the change and retain a checkpoint.
- Resolve stable identity.
- Fetch the permitted current version.
- Parse and classify under a versioned contract.
- Chunk with provenance and access metadata.
- Upsert with retry-safe controlled keys.
- Reconcile access, deletion and supersession.
- Expose freshness, failure and recovery.
The aim is retry-safe, observable reconciliation - not exactly-once delivery, immediate freshness, universal deduplication or automatic permission correctness.
In this article
- Discover the source change
- Resolve stable identity
- Fetch the permitted current version
- Parse and classify
- Chunk with source metadata
- Upsert idempotently
- Reconcile deletion, permission and supersession
- Expose freshness and failure state
1. Discover the source change
Decision: which signal starts work, and what proves its scope is still covered?
Use an initial scan, then a checkpoint. Drive search, user/shared-drive change logs, page tokens and shared-drive parameters have distinct scopes. Record scope, owner and rescan route.
Drive notifications, Notion events, Slack events, GitHub deliveries and Linear webhooks have different renewal, ordering, retry, payload and access limits. Each still needs current state.
Backfill differs: Notion search is not exhaustive or immediate, Slack history is scoped and paginated, and Linear pagination is not an immutable event log.
Illustrative shared-Drive scenario - not an IZZY client case. Rename, move, revision, permission and deletion enter here. An expired channel without a checkpoint sets replay_required, not current.
2. Resolve stable identity
Decision: what key survives presentation changes and addresses every derived record?
A Drive file ID survives a rename and parent change. A copy has its own source identity even when content matches.
Provider identities differ: Notion pages, blocks and data sources; Slack channel and message timestamp; GitHub delivery ID versus path plus ref/SHA; and Linear delivery UUID for a payload, not its object.
| Identity | Controlled meaning | Must not become |
|---|---|---|
| Source record | Native authority: source_system + source_id | Name, URL or delivery |
| Extracted document | Parsed observed source version | Source authority |
| Chunk | Unit linked to source, version and transform | Independent truth |
| Vector/index entry | Store record mapped to chunk_id | Universal provider ID |
| Exact field | Minimum meaning | Boundary |
|---|---|---|
source_system | Provider plus connection/tenant | Do not infer wider coverage |
source_id | Native ID within that scope | Not a display name |
source_version | Provider version observed | unknown when unavailable |
source_updated_at | Provider-reported source update time | Not fetch or index time |
indexed_at | Observed index-write time | Not query-visibility proof |
permission_class | ACL reference for reconciliation | Not store enforcement |
content_hash | Hash under recorded normalisation | Not semantic dedupe |
chunk_id | Addressable chunk key | Links to source/version |
sync_state | Processing/failure state | Not boolean synced |
superseded_by | Replacement pointer | Not automatic |
deleted_at | Deletion/tombstone time | Not downstream proof |
3. Fetch the permitted current version
Decision: can the ingestion identity still read the intended current bytes or object graph?
Drive uses media download for blobs and export for Workspace documents; format and size constrain parser input.
Notion needs current sharing and block traversal; Slack needs token and scopes; GitHub needs path, ref, size and permission; Linear can return HTTP 200 with partial data and errors. Separate fetch from parse: denied is not partial.
A Drive move keeps source_id but may change inherited access. Recheck before serving.
4. Parse and classify
Decision: did the intended document class transform completely under the current contract?
Notion child blocks and unsupported types show why a response is not completeness evidence. n8n’s Default Data Loader loads binary or JSON, attaches metadata and connects a splitter; it does not supply identity, permission reconciliation or parser assurance.
Version the parser/classifier, class, transform and required units. Quarantine unsupported input as schema_parser_failure for replay. A revised brief missing a required section is partial, not current.
5. Chunk with source metadata
Decision: can each chunk be traced, filtered, superseded and removed?
n8n exposes chunk size and overlap; settings do not create identity or prove completeness. Its component map separates loaders, splitters, embeddings, stores and retrievers without recovery.
Pinecone recommends structured IDs and source metadata; Qdrant stores application-supplied JSON payload. Metadata supports filters, not proof of current source permissions.
For a revision, derive addressable chunks, retain the source/version link and mark old chunks for supersession.
6. Upsert idempotently
Decision: can a retry run without accumulating another uncontrolled current representation?
Workflow surfaces differ: n8n’s Pinecone node documents ID-based update; its Qdrant node exposes another surface. Neither defines the provider’s complete API.
n8n Remove Duplicates compares configured workflow fields within input or prior executions; it is not semantic document deduplication. Pinecone upsert overwrites the same record ID but does not remove sibling chunks. Qdrant documents point-specific writes and several synchronisation patterns, not one automatic source guarantee.
Own event, source-version, chunk and store keys. A same-hash Drive copy is a duplicate candidate, not the same authority. Write current entries, link supersession and reconcile siblings.
7. Reconcile deletion, permission and supersession
Decision: what must leave search, become inaccessible or be marked replaced?
Drive’s file resource exposes trash, parent, version, permission and caller-relative capability state; moves can change inherited access. Notion’s trash endpoint changes in_trash, not permanent deletion. Slack deletion events expose channel and deleted timestamp; access still depends on OAuth scope.
Store removal is separate. Pinecone supports ID-, metadata- and namespace deletion with eventual consistency; Qdrant has point deletion. Acceptance is not immediate query absence.
For the Drive brief: rename preserves identity; move requires inherited-access re-evaluation; revision supersedes chunks; a same-content copy is a candidate duplicate; permission loss sets denied; trash/delete sets deleted_at, records a tombstone, removes or quarantines entries and verifies downstream state.
8. Expose freshness and failure state
Decision: can an operator see what is current, what failed and where replay starts?
GitHub’s delivery view is bounded; failures need manual or API redelivery. n8n error workflows need configuration, visibility is access-scoped and failed runs can be retried. Neither is a durable source checkpoint.
sync_state | Observed trigger | Owner action | Clear condition |
|---|---|---|---|
duplicate | Event/key/hash repeat | Classify layer; reconcile | One evidenced current representation |
stale | Source ahead of index evidence | Find lagging stage; replay | Class freshness evidence is current |
denied | Fetch/use not permitted | Stop serving; reconcile | Reprocessing or removal confirmed |
deleted | Trash/delete observed | Tombstone; remove/quarantine | Removal confirmed; no resurrection |
partial | Required units failed | Preserve and isolate | Success or recorded exclusion |
replay_required | Gap, expiry or failure | Replay checkpoint/backfill | New checkpoint; no gap |
schema_parser_failure | Parser cannot produce form | Quarantine; record versions | Compatible reprocessing succeeds |
Measure source observation, source_updated_at, checkpoint, processing and indexed_at separately; set class freshness, not a universal SLA. Only gap-free replay plus reconciled Drive version, access, chunks and removal observably clears replay_required.
Conclusion: make current state an evidenced result
A RAG ingestion pipeline is a controlled lifecycle, not a connector plus synced. Stable identities, reconciliation and observable freshness/failure are the controls. This design has not been live-tested.
Scope one controlled ingestion-pipeline audit
Bring one source, document class, change-volume pattern, access model, freshness expectation and known duplicate or stale-answer failure. IZZY will define what can be observed, reconciled and tested. The decision may be to improve source governance first.
This is exactly the scope of our n8n AI Automation service.
Frequently asked questions
It discovers, identifies, fetches, parses, chunks, upserts, reconciles and exposes freshness or failure.
No. Hashes flag equal content; n8n compares configured fields. Event, source, document, chunk and index duplicates need separate rules.
No. A webhook signals work. Fetch, permission, parse, index, reconciliation and evidence remain separate.
Set denied, stop serving or quarantine, reconcile access, then reprocess or confirm removal. Drive sharing remains an input, not proof.
Record a tombstone and deleted_at, use the provider removal route, prevent replay restoration and confirm absence.
Sources and method
Research was checked on 2026-07-27 using current official Google Drive, Notion, Slack, GitHub, Linear, n8n, Pinecone and Qdrant documentation. Official links sit beside product claims.
This is IZZY architecture guidance. No live tenant, webhook, parser, workflow, index, permission propagation, deletion, replay, retrieval result, SLA or client outcome was tested.