Jeff Mixon / Corpusly: a Google Drive index with no server, no cloud embeddings, and no mirrored files

Created Mon, 13 Jul 2026 00:00:00 -0700 Modified Tue, 29 Sep 2026 02:01:39 -0700
2370 Words
mcp rag typescript embeddings privacy developer-tooling

The premise

Most RAG tools have a moment where your file contents leave your machine, either uploaded to a hosted vector database or sent to a hosted API just to become an embedding. Corpusly is built without that moment. It indexes a Google Drive for semantic search and serves the results to Claude and other MCP clients, and the whole pipeline (fetch, extract, chunk, embed, store, search) runs on the machine that owns the Drive. There is no Corpusly server. The promise isn’t “we don’t log it” or “we delete it afterwards”; there’s simply no vendor system for your data to end up in.

That constraint can sound like a privacy feature bolted onto an ordinary RAG pipeline. It works the other way round. The embedding model, the vector store, how the MCP transport is exposed, even which OAuth scope to request: each of those decisions was made to keep it true.

Updated September 2026. Since this post first went up, the core engine has become free and fair-source (licensed under the Functional Source License, with the source to be published soon), the desktop app has moved into its own project, the extractor registry has grown well past its original 18 formats, and embedding can optionally run on a GPU. The biggest change is how the central claim is backed. It used to rest on a code review, and now a test enforces it. The sections below reflect all of this.

Stack at a glance

  • Core: TypeScript / Node 24+ (a separate Electron + Next.js desktop app drives the same core over a local control API)
  • MCP: @modelcontextprotocol/sdk, the official TypeScript SDK, served over both stdio and loopback streamable HTTP
  • Embeddings: @huggingface/transformers (transformers.js) running nomic-embed-text-v1.5 locally as quantized ONNX, with an opt-in GPU path (CUDA on Linux, DirectML on Windows)
  • Storage: LanceDB for vectors and better-sqlite3 for file-sync state and full-text search, split by access pattern
  • Drive access: googleapis / google-auth-library, directly against the Drive API
  • Extraction: a registry keyed by mimetype that covers dozens of file types, with tesseract.js OCR for scanned PDFs and images

Streaming in, not mirroring

Corpusly never keeps a copy of your Drive on disk. Each file is enumerated, fetched, extracted, chunked and embedded, and only the chunks and vectors are stored; the source bytes are not. The extractor registry picks an extractor by mimetype alone. It covers native Google Docs, Sheets, Slides and Drawings; PDF, with OCR for scanned pages; modern and legacy Office; OpenDocument, EPUB, RTF and HTML; notebooks and SVG/XML/YAML; and email (.eml/.mbox), recursing through attachments with the same registry. Legacy .doc/.xls/.ppt files are read by walking their OLE2/CFB record trees directly instead of shelling out to a converter, and a small hand-written reader handles Word 6.0/95 files that existing libraries reject.

Office and e-book formats share one generic ZIP-of-XML walker rather than a separate library per format. That’s a deliberate choice to keep the number of dependencies touching your file contents small. Files that Drive reports as opaque application/octet-stream are re-typed from their filename extension before dispatch, so the registry doesn’t silently skip them. The registry also carries a generation number, so adding an extractor makes files skipped by an older registry eligible again instead of leaving them skipped forever.

Embeddings that never leave the machine

This is the part that has to be true for everything else to mean anything. Text is chunked with an overlapping sliding window snapped to whitespace and handed to nomic-embed-text-v1.5, a 768-dimension model under the Apache 2.0 license. I chose it over Gemma-licensed alternatives so the redistribution terms wouldn’t work against the product. It loads through transformers.js’s pipeline("feature-extraction", ...), quantized to 8-bit ONNX. Its weights, a few tens of megabytes, are downloaded once from Hugging Face into a directory Corpusly controls and reused for every embedding after that. The embeddings are computed in-process, on the same CPU that runs the rest of Corpusly, or on its GPU if you opt in (more below). Queries and documents get the model’s own asymmetric prefixes (search_query: / search_document:), so each side of a retrieval is embedded for the role it actually plays.

The first version of this post pointed at the code as evidence. The embedding step sat behind a provider interface whose hosted option was an unimplemented stub, and a grep for cloud-embedding endpoints found nothing. That was true, but “I grepped and found nothing” is weak proof, and it only covers the day you ran the grep. The stub has since been deleted, so local is the only embedding provider the config accepts, and a test now checks the claim.

Proving it, not asserting it

The egress test runs the real pipeline, indexing a Drive and then searching it, with Node’s socket layer trapped underneath. It hooks net.Socket.prototype.connect rather than fetch, because every HTTP client in the process ends up there: undici, the http/https modules, and raw net.connect calls alike. Each connection attempt is recorded and checked against a short allowlist of the hosts Corpusly is documented to contact:

  • loopback
  • Google’s APIs (your own Drive, and OAuth token refresh)
  • Hugging Face, for the embedding model on first run
  • the CDN that serves Tesseract’s language data, on first OCR

Corpusly has no server of its own, so no Corpusly host appears on that list. The optional GPU runtime downloads from two more hosts, and the test also checks that neither is contacted on the default CPU path.

Two details keep the test honest. Before measuring anything, it tries to reach a host that isn’t allowed and fails unless the trap catches the attempt, so a trap that has silently stopped working can’t produce a green result. It also caught a real bug in its own first version, where one calling convention for net.connect slipped past the hook. The test blocks CI, alongside a static-analysis rule that forbids network imports anywhere in the extraction, chunking and storage code. If “nothing leaves the machine” ever stops being true, the build breaks that day.

What “local” actually costs

Running your own embedding model means inheriting the memory management a hosted provider would otherwise handle for you. Transformer attention scales with batch size times the square of the sequence length. Early on, one dense file of about 150 chunks, batched together for throughput, pushed a single ONNX call to 32 GB and killed the process. Sub-batching plus per-chunk token truncation fixed it: peak memory dropped to 4.4 GB, and because the machine stopped thrashing, it also ran faster than the naive version. That’s the honest cost of keeping everything local. The machine has to be able to do the work, and you may find out the hard way first.

Later passes pushed the same idea further. Chunking, embedding and storing now stream through fixed 256-chunk windows, so peak memory depends on the window size rather than on the largest file in your Drive. OCR runs in a separate process, a change made after one malformed TIFF crashed the image library in-process and trapped indexing in a restart loop at 93%. A file that keeps crashing is quarantined instead of retried forever. OCR output is now kept or dropped based on Tesseract’s own confidence score instead of how much text came back, because a photo of a hedge “reads” as thousands of junk characters while a real receipt might be 68.

When the CPU isn’t enough

For large Drives, embedding can optionally move to a GPU. On Linux that’s CUDA, through a hash-pinned runtime pack that’s downloaded only when you turn the feature on. On Windows it’s DirectML, steered to the discrete card on laptops that have two GPUs. Batch size is set from the VRAM it detects, and if the GPU crashes or runs out of memory, a breaker falls back to the CPU instead of failing the index. On a laptop RTX 5080 it measured about 15× faster, from roughly 11 to 170 chunks a second. Switching devices never forces a re-embed: the device isn’t part of an embedding’s identity, since the same model produces the same vectors either way.

Two stores, split by access pattern

Vectors and file-sync state live in different databases on purpose. LanceDB holds an Arrow-schema table of chunk vectors, opened so a reader always sees the writer’s latest commit, and searched with cosine-distance nearest-neighbor queries. Everything else, including file sync cursors, per-file embedding status and a full-text index, lives in SQLite via better-sqlite3. Approximate nearest-neighbor search over high-dimensional vectors is a different problem from exact keyword lookup and transactional state, so the two don’t share a store just because one box would look simpler on a diagram.

That split is also what makes a newly connected Drive searchable right away instead of after an overnight indexing run. Cataloging a Drive records every file as pending within seconds, using metadata only, with no embedding yet. A search hits the SQLite full-text index, which covers pending files too, then embeds any top results that aren’t embedded yet before answering. A background drain fills in the rest over time. Hybrid search blends that keyword layer with vector similarity, so the first query against a huge Drive returns something useful instead of a “still indexing” message.

Serving it over MCP

The MCP server is built on the official @modelcontextprotocol/sdk and registers four core tools with zod-typed inputs:

  • search_drive, for semantic search that returns ranked chunks with citation links
  • get_file_content, for a file’s full reconstructed text
  • list_indexed_files, for a paginated catalog
  • resync, to report how much is embedded versus still pending

All four run against the local pipeline described above, so nothing an MCP client asks for passes through a Corpusly server.

The server is reachable over two transports. Claude Desktop and Claude Code launch it over stdio, and a cross-process write lock means exactly one running instance embeds and writes while any others serve read-only. A separate HTTP transport lets a background daemon serve the same tools continuously. It binds to 127.0.0.1 only, requires a bearer token, and checks the Host and Origin headers before authentication even runs. It deliberately skips TLS, because encrypting a loopback socket doesn’t protect against anything a same-host attacker couldn’t already reach. A small proxy transport lets a client that only speaks stdio ride that same HTTP session, so the spawned process and the long-running daemon end up talking to one server rather than two. (Building on an existing Drive MCP server was the other option, but none of them exposed an incremental change feed, which a continuously synced index needs. This one talks to the Drive API directly.)

One command now registers Corpusly with Claude Desktop, Claude Code, OpenClaw or Cursor. It backs up the client’s config and writes the new one atomically, so a crash partway through can’t leave a broken file behind.

ChatGPT is the one documented exception to local-only, and it stays off unless you turn it on. ChatGPT’s connectors call in from OpenAI’s servers, so they can’t be served over loopback. When you opt in, Corpusly starts a separate public MCP listener that you expose through a tunnel you run yourself. It implements OpenAI’s search/fetch tool contract and sits behind a small OAuth 2.1 authorization server built into Corpusly, with dynamic client registration, PKCE, and a consent window the owner has to open before any client can connect. Results still come from the same local index, but they go to whatever ChatGPT asked for them. That difference is why the feature is opt-in and labeled as the exception.

The constraint that shapes everything downstream

MCP’s stdio transport uses stdout as the wire protocol itself. Every JSON-RPC message is a line of stdout, so one stray console.log anywhere in the dependency tree corrupts the stream and silently drops the connection. Trusting every dependency, including transitive ones added later, never to write to stdout isn’t a bet worth making. Instead, a small shim routes noisy libraries’ console output to a stderr logger. pdf.js in particular floods console.warn while parsing. The rule is enforced once, in one place, rather than by convention in every file.

The background daemon that keeps all this running rests on two separate locks, one for “at most one daemon” and one for “at most one writer,” plus a priority mutex so an interactive search never waits behind a bulk re-embedding pass. Writes are also fenced by a token issued each time the writer lock is acquired. A process that lost the lock partway through a write can’t pass itself off as the current writer, and neither can a recycled process ID that happens to match an old owner. None of that is visible from the MCP client’s side. It follows the same principle as the transport proxy: however a client is configured to reach Corpusly, exactly one process is doing the writing.

Local-first as a compliance property, not just a pitch

Indexing an entire Drive requires Google’s restricted drive.readonly scope. The narrower drive.file scope is per-file and doesn’t cascade to folder contents, so it can’t do this job. Restricted scopes normally mean Google’s own costly third-party security assessment, but that assessment is triggered by handling the data on a server. Corpusly qualifies for Google’s local-client exemption instead, because there’s literally no server to assess. Designing for privacy turned out to be the cheaper compliance path too, and I’m glad I got there by taking the architecture seriously rather than working backwards from the paperwork. Working through the exemption properly also closed a handful of real gaps in the code: an OAuth state check, revoking the token on disconnect, a purge command, private file permissions, and limits against zip bombs.

Closer

None of this adds up to a single privacy feature. It’s what follows from refusing to add a server in the first place. Every step from a Drive API call to a searchable vector runs on the machine that asked for it, there’s no hosted embedding path to fall back to, and a test fails the build if anything starts talking to a host it shouldn’t.

The core engine is free under the Functional Source License, with each release converting to Apache 2.0 two years later, and the source will be published soon. The desktop app is in early access, and the waitlist is open at corpusly.ai.