# Corpusly: a Google Drive index with no server, no cloud embeddings, and no mirrored files

> Corpusly indexes a Google Drive for semantic search and serves it to Claude and other MCP clients, with no vendor server in the path your documents take. How the local embedding pipeline and the two-transport MCP server work, and what it took to make 'nothing leaves the machine' true: a memory blowout fixed by sub-batching, a stdout-corruption problem solved in one place, an OAuth scope decision that doubled as a compliance strategy, and a CI test that holds every outbound connection to an allowlist.

By Jeff Mixon, published 2026-07-13. Canonical: https://www.jeffmixon.com/post/corpusly-local-first-rag-mcp/

## The premise

Most RAG tools have a moment where your file contents leave your machine,
either uploaded to a hosted vector database or sent to a hosted API just to
become an embedding. Corpusly is built without that moment. It indexes a
Google Drive for semantic search and serves the results to Claude and other
MCP clients, and the whole pipeline (fetch, extract, chunk, embed, store,
search) runs on the machine that owns the Drive. There is no Corpusly server.
The promise isn't "we don't log it" or "we delete it afterwards"; there's
simply no vendor system for your data to end up in.

That constraint can sound like a privacy feature bolted onto an ordinary RAG
pipeline. It works the other way round. The embedding model, the vector
store, how the MCP transport is exposed, even which OAuth scope to request:
each of those decisions was made to keep it true.

*Updated September 2026.* Since this post first went up, the core engine has
become free and fair-source (licensed under the Functional Source License,
with the source to be published soon), the desktop app has moved into its own
project, the extractor registry has grown well past its original 18 formats,
and embedding can optionally run on a GPU. The biggest change is how the
central claim is backed. It used to rest on a code review, and now a test
enforces it. The sections below reflect all of this.

## Stack at a glance

- **Core:** TypeScript / Node 24+ (a separate Electron + Next.js desktop app
  drives the same core over a local control API)
- **MCP:** `@modelcontextprotocol/sdk`, the official TypeScript SDK, served
  over both stdio and loopback streamable HTTP
- **Embeddings:** `@huggingface/transformers` (transformers.js) running
  `nomic-embed-text-v1.5` locally as quantized ONNX, with an opt-in GPU path
  (CUDA on Linux, DirectML on Windows)
- **Storage:** LanceDB for vectors and `better-sqlite3` for file-sync state
  and full-text search, split by access pattern
- **Drive access:** `googleapis` / `google-auth-library`, directly against the
  Drive API
- **Extraction:** a registry keyed by mimetype that covers dozens of file
  types, with `tesseract.js` OCR for scanned PDFs and images

## Streaming in, not mirroring

Corpusly never keeps a copy of your Drive on disk. Each file is enumerated,
fetched, extracted, chunked and embedded, and only the chunks and vectors are
stored; the source bytes are not. The extractor registry picks an extractor
by mimetype alone. It covers native Google Docs, Sheets, Slides and Drawings;
PDF, with OCR for scanned pages; modern and legacy Office; OpenDocument,
EPUB, RTF and HTML; notebooks and SVG/XML/YAML; and email (`.eml`/`.mbox`),
recursing through attachments with the same registry. Legacy
`.doc`/`.xls`/`.ppt` files are read by walking their OLE2/CFB record trees
directly instead of shelling out to a converter, and a small hand-written
reader handles Word 6.0/95 files that existing libraries reject.

Office and e-book formats share one generic ZIP-of-XML walker rather than a
separate library per format. That's a deliberate choice to keep the number of
dependencies touching your file contents small. Files that Drive reports as
opaque `application/octet-stream` are re-typed from their filename extension
before dispatch, so the registry doesn't silently skip them. The registry also
carries a generation number, so adding an extractor makes files skipped by an
older registry eligible again instead of leaving them skipped forever.

## Embeddings that never leave the machine

This is the part that has to be true for everything else to mean anything.
Text is chunked with an overlapping sliding window snapped to whitespace and
handed to `nomic-embed-text-v1.5`, a 768-dimension model under the Apache 2.0
license. I chose it over Gemma-licensed alternatives so the redistribution
terms wouldn't work against the product. It loads through transformers.js's
`pipeline("feature-extraction", ...)`, quantized to 8-bit ONNX. Its weights,
a few tens of megabytes, are downloaded once from Hugging Face into a
directory Corpusly controls and reused for every embedding after that. The
embeddings are computed in-process, on the same CPU that runs the rest of
Corpusly, or on its GPU if you opt in (more below). Queries and documents get
the model's own asymmetric prefixes (`search_query:` / `search_document:`),
so each side of a retrieval is embedded for the role it actually plays.

The first version of this post pointed at the code as evidence. The embedding
step sat behind a provider interface whose hosted option was an unimplemented
stub, and a grep for cloud-embedding endpoints found nothing. That was true,
but "I grepped and found nothing" is weak proof, and it only covers the day
you ran the grep. The stub has since been deleted, so `local` is the only
embedding provider the config accepts, and a test now checks the claim.

## Proving it, not asserting it

The egress test runs the real pipeline, indexing a Drive and then searching
it, with Node's socket layer trapped underneath. It hooks
`net.Socket.prototype.connect` rather than `fetch`, because every HTTP client
in the process ends up there: undici, the `http`/`https` modules, and raw
`net.connect` calls alike. Each connection attempt is recorded and checked
against a short allowlist of the hosts Corpusly is documented to contact:

- loopback
- Google's APIs (your own Drive, and OAuth token refresh)
- Hugging Face, for the embedding model on first run
- the CDN that serves Tesseract's language data, on first OCR

Corpusly has no server of its own, so no Corpusly host appears on that list.
The optional GPU runtime downloads from two more hosts, and the test also
checks that neither is contacted on the default CPU path.

Two details keep the test honest. Before measuring anything, it tries to
reach a host that isn't allowed and fails unless the trap catches the
attempt, so a trap that has silently stopped working can't produce a green
result. It also caught a real bug in its own first version, where one calling
convention for `net.connect` slipped past the hook. The test blocks CI,
alongside a static-analysis rule that forbids network imports anywhere in the
extraction, chunking and storage code. If "nothing leaves the machine" ever
stops being true, the build breaks that day.

## What "local" actually costs

Running your own embedding model means inheriting the memory management a
hosted provider would otherwise handle for you. Transformer attention scales
with batch size times the square of the sequence length. Early on, one dense
file of about 150 chunks, batched together for throughput, pushed a single
ONNX call to 32 GB and killed the process. Sub-batching plus per-chunk token
truncation fixed it: peak memory dropped to 4.4 GB, and because the machine
stopped thrashing, it also ran faster than the naive version. That's the
honest cost of keeping everything local. The machine has to be able to do the
work, and you may find out the hard way first.

Later passes pushed the same idea further. Chunking, embedding and storing now
stream through fixed 256-chunk windows, so peak memory depends on the window
size rather than on the largest file in your Drive. OCR runs in a separate
process, a change made after one malformed TIFF crashed the image library
in-process and trapped indexing in a restart loop at 93%. A file that keeps
crashing is quarantined instead of retried forever. OCR output is now kept or
dropped based on Tesseract's own confidence score instead of how much text
came back, because a photo of a hedge "reads" as thousands of junk characters
while a real receipt might be 68.

## When the CPU isn't enough

For large Drives, embedding can optionally move to a GPU. On Linux that's
CUDA, through a hash-pinned runtime pack that's downloaded only when you turn
the feature on. On Windows it's DirectML, steered to the discrete card on
laptops that have two GPUs. Batch size is set from the VRAM it detects, and if
the GPU crashes or runs out of memory, a breaker falls back to the CPU instead
of failing the index. On a laptop RTX 5080 it measured about 15× faster, from
roughly 11 to 170 chunks a second. Switching devices never forces a re-embed:
the device isn't part of an embedding's identity, since the same model
produces the same vectors either way.

## Two stores, split by access pattern

Vectors and file-sync state live in different databases on purpose. LanceDB
holds an Arrow-schema table of chunk vectors, opened so a reader always sees
the writer's latest commit, and searched with cosine-distance nearest-neighbor
queries. Everything else, including file sync cursors, per-file embedding
status and a full-text index, lives in SQLite via `better-sqlite3`. Approximate
nearest-neighbor search over high-dimensional vectors is a different problem
from exact keyword lookup and transactional state, so the two don't share a
store just because one box would look simpler on a diagram.

That split is also what makes a newly connected Drive searchable right away
instead of after an overnight indexing run. Cataloging a Drive records every
file as pending within seconds, using metadata only, with no embedding yet. A
search hits the SQLite full-text index, which covers pending files too, then
embeds any top results that aren't embedded yet before answering. A
background drain fills in the rest over time. Hybrid search blends that
keyword layer with vector similarity, so the first query against a huge Drive
returns something useful instead of a "still indexing" message.

## Serving it over MCP

The MCP server is built on the official `@modelcontextprotocol/sdk` and
registers four core tools with zod-typed inputs:

- `search_drive`, for semantic search that returns ranked chunks with
  citation links
- `get_file_content`, for a file's full reconstructed text
- `list_indexed_files`, for a paginated catalog
- `resync`, to report how much is embedded versus still pending

All four run against the local pipeline described above, so nothing an MCP
client asks for passes through a Corpusly server.

The server is reachable over two transports. Claude Desktop and Claude Code
launch it over stdio, and a cross-process write lock means exactly one
running instance embeds and writes while any others serve read-only. A
separate HTTP transport lets a background daemon serve the same tools
continuously. It binds to `127.0.0.1` only, requires a bearer token, and
checks the Host and Origin headers before authentication even runs. It
deliberately skips TLS, because encrypting a loopback socket doesn't protect
against anything a same-host attacker couldn't already reach. A small proxy
transport lets a client that only speaks stdio ride that same HTTP session,
so the spawned process and the long-running daemon end up talking to one
server rather than two. (Building on an existing Drive MCP server was the
other option, but none of them exposed an incremental change feed, which a
continuously synced index needs. This one talks to the Drive API directly.)

One command now registers Corpusly with Claude Desktop, Claude Code, OpenClaw
or Cursor. It backs up the client's config and writes the new one atomically,
so a crash partway through can't leave a broken file behind.

ChatGPT is the one documented exception to local-only, and it stays off
unless you turn it on. ChatGPT's connectors call in from OpenAI's servers, so
they can't be served over loopback. When you opt in, Corpusly starts a
separate public MCP listener that you expose through a tunnel you run
yourself. It implements OpenAI's `search`/`fetch` tool contract and sits
behind a small OAuth 2.1 authorization server built into Corpusly, with
dynamic client registration, PKCE, and a consent window the owner has to open
before any client can connect. Results still come from the same local index,
but they go to whatever ChatGPT asked for them. That difference is why the
feature is opt-in and labeled as the exception.

## The constraint that shapes everything downstream

MCP's stdio transport uses stdout as the wire protocol itself. Every JSON-RPC
message is a line of stdout, so one stray `console.log` anywhere in the
dependency tree corrupts the stream and silently drops the connection.
Trusting every dependency, including transitive ones added later, never to
write to stdout isn't a bet worth making. Instead, a small shim routes noisy
libraries' console output to a stderr logger. `pdf.js` in particular floods
`console.warn` while parsing. The rule is enforced once, in one place, rather
than by convention in every file.

The background daemon that keeps all this running rests on two separate
locks, one for "at most one daemon" and one for "at most one writer," plus a
priority mutex so an interactive search never waits behind a bulk re-embedding
pass. Writes are also fenced by a token issued each time the writer lock is
acquired. A process that lost the lock partway through a write can't pass
itself off as the current writer, and neither can a recycled process ID that
happens to match an old owner. None of that is visible from the MCP client's
side. It follows the same principle as the transport proxy: however a client
is configured to reach Corpusly, exactly one process is doing the writing.

## Local-first as a compliance property, not just a pitch

Indexing an entire Drive requires Google's restricted `drive.readonly` scope.
The narrower `drive.file` scope is per-file and doesn't cascade to folder
contents, so it can't do this job. Restricted scopes normally mean Google's
own costly third-party security assessment, but that assessment is triggered
by handling the data on a server. Corpusly qualifies for Google's
local-client exemption instead, because there's literally no server to
assess. Designing for privacy turned out to be the cheaper compliance path
too, and I'm glad I got there by taking the architecture seriously rather
than working backwards from the paperwork. Working through the exemption
properly also closed a handful of real gaps in the code: an OAuth `state`
check, revoking the token on disconnect, a purge command, private file
permissions, and limits against zip bombs.

## Closer

None of this adds up to a single privacy feature. It's what follows from
refusing to add a server in the first place. Every step from a Drive API call
to a searchable vector runs on the machine that asked for it, there's no
hosted embedding path to fall back to, and a test fails the build if anything
starts talking to a host it shouldn't.

The core engine is free under the Functional Source License, with each
release converting to Apache 2.0 two years later, and the source will be
published soon. The desktop app is in early access, and the waitlist is open
at [corpusly.ai](https://corpusly.ai).