A Graph You Can Audit: Provenance-First Agent Memory

Why combine graph and vector retrieval for agent memory?

Vector retrieval answers what looks similar; graph retrieval answers how records are connected and why. Combining them helps an agent discover relevant language while preserving explicit evidence for relations such as supersedes, verifies, contradicts, or derives from.

A vector can tell you that two notes discuss authentication. It cannot tell you that one supersedes the other, that both came from the same migration, or that a fix was verified by a particular test. Those are relations with evidence and direction, not distances in an embedding space.

For engineering memory, collapsing the two concepts creates confident but unauditable retrieval. The graph layer therefore accepts only explicit relation records with a type, source, target, reason, confidence, source agent, and tenant-project-repository scope.

The schema makes stale vectors impossible to mistake for current ones

Each memory may hold a 384-dimensional embedding plus the embedding model, embedded timestamp, and the content hash used to produce it. A database constraint requires the embedding and its provenance to be either complete and current or entirely absent.

A trigger clears the embedding whenever the memory content hash changes. This moves freshness enforcement below application code. Even if a caller forgets to schedule re-embedding, retrieval cannot quietly present an old vector as if it represented the new text.

Relations are tenant-scoped at every boundary

The relation table carries tenant, project, and repository identity in addition to source and target memory IDs. Foreign keys protect endpoint integrity, a check forbids self-edges, confidence is constrained to the zero-to-one range, and a compound unique index prevents duplicate typed edges in one scope.

Service methods authorize both endpoints and reject cross-scope links. Incoming and outgoing indexes support bounded traversal by relation type. The graph API does not get to infer tenancy from node IDs after the fact.

I chose local embeddings to keep text at the trust boundary

The embedding authority is the pinned Xenova all-MiniLM-L6-v2 model running locally at 384 dimensions. That avoids sending memory content to another inference service and makes the model and dimension explicit migration concerns.

Changing the model is not a configuration toggle. It requires backfilling vectors and rebuilding the corresponding retrieval index. The service exposes a bounded authorized backfill operation, validates vector dimensions, and records the model and content hash beside every result.

Hybrid search is an enhancement, never the only way home

A query is embedded locally and sent to a vector-first hybrid retrieval endpoint with a 0.7 vector and 0.3 graph weighting. The service filters results back to memory nodes, hydrates authorized records, and folds them into the existing ranking path.

If hybrid retrieval is unavailable, the adapter falls back to vector search. If semantic infrastructure is disabled or fails, deterministic PostgreSQL substring and source-file matching still works. A full-text GIN index exists, but the current ORM path does not query it. This graceful degradation was a design requirement because memory recall sits on the critical path of every agent session.

The graph was built from evidence, not from cosine proximity

The migration imported 63 memories and embedded 62 active records; one deleted record intentionally remained unembedded. Sixteen RELATED_TO edges were derived only when two memories shared recorded source provenance. The derivation is deterministic and idempotent.

This produces a smaller graph than automatic semantic linking, but every edge can answer the question: why are these records connected? For operational memory, a sparse explainable graph is safer than a dense attractive one.

Verification tested both retrieval modes

The target platform reported graph, vector, and hybrid retrieval ready. A runtime smoke query returned five project-repository scoped hybrid hits. A second test selected a known provenance-linked pair and required the graph path endpoint to traverse that imported edge.

Unit tests also forced the hybrid endpoint to fail and confirmed vector fallback. A fresh local run passed 48 tests across the core, vault, synchronization, authentication, and semantic-retrieval packages. Eight opt-in database integration tests were skipped in that run, so they are not included in the passing count.

Where graph retrieval should go next

The next useful edge types are not more synonyms. They are operational semantics: supersedes, contradicts, verifies, caused-by, and derived-from. Each needs a clear creation authority and a testable rule, because direction and lifecycle matter.

Ranking should then be evaluated against real agent decisions. A graph path is valuable only if it brings the correct invariant or fix into context sooner. Offline recall sets, human relevance labels, and post-run usefulness signals should decide whether graph weight rises above 0.3, not aesthetic preference for connected diagrams.