Gading's Hideout
🇺🇸

How I Build Agent Memory Free from Vendor Lock-in

How I restructured my AI Agent's (Nouva) memory system from a noisy, bloated Vector DB into a lean, Two-Tier Hybrid RAG + NAS Markdown setup that covers 95% of personal needs.

Jul 05, 2026
•
7 min read

In my previous post, The GraphRAG Trap: Why I Uninstalled Neo4j from My Personal Assistant, I shared my journey of stripping down Neo4j and Graphitti from my AI assistant, Nouva. The verdict was simple: it was overkill, added massive maintenance overhead, and burned through LLM tokens for relationship traversals that just weren't necessary for a single-user setup.

But after uninstalling Neo4j, I hit a new bottleneck: How do I manage Nouva's long-term chat logs and history so they remain searchable and fast, without bloating the RAG database?

If you blindly dump raw daily chat logs (YYYY-MM-DD-HHMM.md) into a standard Vector DB (RAG) like AnythingLLM, you quickly fall into a different pitfall: Vector Dilution & Noise.

Today, I’ll share how I designed a hybrid memory system that is lightweight, requires zero extra database dependencies, and covers 95% of my personal assistant needs.

The Classic RAG Problem: "Vector Dilution"

When I first transitioned to a conventional RAG setup, I synced every raw daily log file to the vector database. It was a disaster.

A single daily chat log is a noisy mess. It contains deep Kubernetes architecture planning, casual banter about Tifa in FF7 Rebirth, system error logs, and title reminders from my wife to buy skincare. When this file is chunked (split into 1000-character blocks) and embedded, the semantic vector representation gets heavily diluted.

The side effect was frustrating. If I asked: "Va, do you remember our discussion about ESG yesterday?", the RAG would return a chunk containing skincare reminders because of some random keyword alignment within the same daily file. The actual context I needed was lost in the noise.

Why We Still Do NOT Need mem0 or GraphRAG

Before diving into the solution, let me reinforce why I believe 95% of personal AI setups do not need complex memory frameworks:

  1. GraphRAG (99% Unnecessary): The indexing process is extremely token-hungry and hammers local CPU/GPU resources. You don't need a heavy graph database to infer that a server downtime on June 24th is related to a specific config change.
  2. mem0 (90% Unnecessary): mem0 extracts dynamic facts into a separate vector/graph database. However, we already have a static MEMORY.md file in our local workspace containing curated facts (PC specs, family info, active projects). A markdown file is infinitely easier to edit and debug manually compared to cleaning up misidentified nodes in a hidden graph database.

The remaining 5% of use cases are only relevant if you want to run high-level, long-term abstract synthesis (e.g., "Plot my mood changes over the past year based on our chats"). For a developer's daily workflow? It's useless.

The Solution: Two-Tier Hybrid RAG + NAS Markdown

TL;DR: I built the RAG here as a memory index only. Every incoming keyword will return several candidate daily log dates (in YYYY-MM-DD format). The actual memory is purely markdown file-based, with the file name being that date. So after the LLM gets the candidate dates from the index, it directly accesses the markdown file to read the original content.

Instead, I implemented a Two-Tier Hybrid Memory architecture that separates the indexing layer from the raw storage layer:

[User Query] 
     │
     ▼
[Unified query-memory.py]
     │
     ├─► Step 1: Semantic Search (RAG via AnythingLLM)
     │           Searches MEMORY_INDEX.md (The Map)
     │           Result: Finds Target Date (e.g., 2026-06-24)
     │
     ├─► Step 2: Direct Read (NAS Zip Archive)
     │           Reads raw transcript directly from zip on NAS
     │
     └─► Fallback: Fast Keyword Search (Grep in NAS Zip)
                   If RAG confidence is low or no matches found

Tier 1: Semantic Index Map (MEMORY_INDEX.md)

We no longer sync long, noisy raw chat transcripts to the vector database. Instead, we only upload two clean files to AnythingLLM:

  1. MEMORY.md (Curated long-term facts).
  2. MEMORY_INDEX.md (The navigation map).

The MEMORY_INDEX.md is generated automatically by a weekly sync script. It contains a highly summarized list of topics per date:

### 📅 Wednesday, June 24, 2026 (2026-06-24)
  - **🚀 AnythingLLM Migration & Configuration**
  - **🛠️ Proxmox & Terraform State Alignment**
  - **Discussions (Chat)**
    - but I'm curious what this ESG is chasing? is it full of woke people?

Because the file is clean and free of system error stack traces, the RAG embeds it perfectly using a local bge-m3 model running on the homelab's CPU (Ryzen 7 5825U).

Tier 2: Raw Archive (NAS Zip)

All raw session logs (YYYY-MM-DD-HHMM.md) are zipped and transferred to our NAS (virtual-nas) after a 2-day grace period, keeping the local agent host disk footprint minimal.

Tier 3: The Glue (Unified Script)

I wrote a unified wrapper script query-memory.py to bridge the two layers:

  1. When I ask: "Do you remember our discussion about ESG?", the script queries the RAG (AnythingLLM).
  2. The RAG matches the query to the MEMORY_INDEX.md chunk and returns the date: 2026-06-24.
  3. The script automatically opens the corresponding ZIP archive on the NAS, reads the raw 2026-06-24.md transcript directly into RAM, and feeds it to the AI Agent.
  4. Fallback: If the semantic search returns low confidence, the script falls back to a fast regex keyword search across all zipped markdown files on the NAS.

Why This Approach Wins

  • Zero Extra Dependencies: It runs on standard Python libraries (zipfile & requests). No extra database containers or services to maintain.
  • Resource-Friendly: The local agent host (running on a constrained 4GB RAM LXC) stays incredibly light because embedding workloads are delegated to Ollama in the backend.
  • 100% Portable: Because the data is stored in raw Markdown and standard ZIPs, it is completely vendor-agnostic. If I want to migrate my assistant to another LLM platform, I can import the NAS archives in seconds.
  • Vendor Lock-in Free (Honestly, I'm Tired of It): Many modern memory frameworks force you to use their proprietary cloud databases, paid APIs, or custom formats that keep your data hostage. Honestly, I'm tired of being locked into a single ecosystem. With raw .md and standard .zip files, the data is 100% mine. If tomorrow I want to switch LLM providers or migrate from OpenClaw to another framework, I just copy-paste the zip files without worrying about compatibility.
  • Transparent & Debuggable: If the AI remembers something incorrectly, I can simply open the markdown file and edit the line. No vector database queries or graph node deletions required.

Keep it simple. Sometimes, the best solution to complex AI infrastructure problems is to step back and use structured text files glued together with precise automation scripts.

Lesson Learned: Back to Basics

From this cycle of building and tearing down my memory stack, there's one valuable lesson: Don't get blinded by the hype.

The AI landscape moves at a breakneck pace. Every week, there's a new library or framework promising the world—GraphRAG, mem0, autonomous memory layers, you name it. It's incredibly easy to fall into the trap of "architectural FOMO" and install everything just because it looks cool on paper or on your social feeds.

But before running that install command, ask yourself:

  • Do I actually need this? (A personal assistant use case is vastly different from a multi-tenant corporate tool).
  • Do I understand the core concept? (Do I just need date mapping and keyword lookup, or do I genuinely need complex graph database traversals?).

Going back to the basics and understanding the core mechanics of the problem you are trying to solve is far more valuable than stacking dependencies. Sometimes, the most robust solution is just a structured text file (Markdown), a bit of regex, and a simple, precise automation script. It saves tokens, saves RAM, and saves you from maintenance headaches down the road.

What's Next?

This flat RAG + NAS scheme is a solid starting foundation. However, as the daily notes accumulate over hundreds of days, the next challenges are tackling index bloating, vector dilution, and handling graph traversals (linking threaded chats across days) without sacrificing performance.

In the next post, Building Agent Memory with Vendor Lock-in Resistance (Part 2), I will dissect the continuation of this architecture. We will discuss how to refactor Nouva's memory system using a Hierarchical Summary and a simple Hybrid Scoring approach to keep memory retrieval instant, tidy, and of course, 100% free from vendor lock-in.

How do you handle your AI assistant's memory? Let's discuss on Threads.