The premise
Most RAG tools have a moment where your file contents leave your machine: uploaded to a hosted vector database, or sent to a hosted API just to become an embedding. Corpusly is built to not have that moment. It indexes a Google Drive for semantic search and serves the result to Claude and other MCP clients, and the whole pipeline — fetch, extract, chunk, embed, store, search — runs on the machine that owns the Drive. There’s no Corpusly server. Not “we don’t log it” or “we delete it after”: there is genuinely nowhere else for the data to go.
That constraint reads like a privacy feature bolted onto a normal RAG pipeline. It’s closer to the reverse: which embedding model, which vector store, how the MCP transport gets exposed, even which OAuth scope to request — every one of those decisions got made in service of keeping it true.
Stack at a glance
- Core: TypeScript / Node 20+, npm workspaces (a separate Electron + Next.js desktop app shares the same core)
- MCP:
@modelcontextprotocol/sdk, the official TypeScript SDK, served over both stdio and loopback streamable HTTP - Embeddings:
@huggingface/transformers(transformers.js) runningnomic-embed-text-v1.5locally, as quantized ONNX - Storage: LanceDB for vectors,
better-sqlite3for file-sync state and full-text search — split by access pattern - Drive access:
googleapis/google-auth-library, direct against the Drive API - Extraction: an 18-format registry with
tesseract.jsOCR fallback for scanned PDFs
Streaming in, not mirroring
Corpusly never syncs a copy of your Drive to disk. A file is
enumerated, fetched, extracted, chunked, and embedded, and only the
chunks and vectors persist — not the source bytes. The extractor
registry dispatches purely on mimetype across 18 formats: native
Google Docs/Sheets/Slides, PDF (with OCR fallback for scanned
pages), modern and legacy Office (.doc/.xls/.ppt read via
direct OLE2/CFB record-tree walks, not a converter shelling out to
something else), OpenDocument, EPUB, RTF, HTML, and email
(.eml/.mbox, recursing through attachments via the same
registry). Office and e-book formats share one generic ZIP-of-XML
walker instead of a bespoke library per format — a deliberate bias
toward fewer dependencies touching your file contents, not more.
Files Drive reports as opaque application/octet-stream get
re-typed from their filename extension before dispatch, so the
registry doesn’t silently skip them.
Embeddings that never leave the machine
This is the part that has to actually be true for the rest of the
pitch to mean anything. Text is chunked with a sliding window
(overlap, snapped to whitespace boundaries) and handed to
nomic-embed-text-v1.5, an Apache-2.0-licensed, 768-dimension
model, chosen deliberately over Gemma-licensed alternatives so the
redistribution terms wouldn’t fight the product. It loads through
transformers.js’s pipeline("feature-extraction", ...), quantized
to 8-bit ONNX, with its weights cached in a directory Corpusly
controls — downloaded once from Hugging Face, a few tens of
megabytes, then reused for every embedding after that, computed
in-process on the same CPU already running the rest of Corpusly.
Query and document text get the model’s own asymmetric prefixes
(search_document: / search_query:), so the two sides of a
retrieval are embedded for the roles they actually play.
The detail that convinced me this wasn’t just a default worth
trusting: the embedding step sits behind a provider interface
(local or voyage, picked in config), and the hosted path is a
nine-line stub that just throws “not yet implemented.” There’s no
quiet cloud fallback wired up behind a flag — the interface exists
so a hosted backend could plug in later without a rewrite, but as
shipped, there is no code path that sends chunk text anywhere to
become a vector. A grep for cloud-embedding endpoints across the
source turns up nothing; the only outbound network calls anywhere
in the codebase are Drive API and OAuth traffic, fetching content,
never embedding it.
What “local” actually costs
Running your own embedding model instead of calling someone else’s means you inherit their memory-management problem. Transformer attention scales with batch size times sequence length squared, and early on, one dense file with roughly 150 chunks, batched together for throughput, blew a single ONNX call up to 32 GB and took the process out. The fix was sub-batching plus per-chunk token truncation, which dropped peak memory to 4.4 GB and, as a side effect of not thrashing, ran faster than the naive version had. That’s the honest tradeoff behind “nothing leaves the machine”: the machine has to actually be able to do the work, and finding that out costs you an OOM crash before it costs you anything else.
Two stores, split by access pattern
Vectors and file-sync state live in different databases on purpose.
LanceDB holds an Arrow-schema table of chunk vectors, opened so a
reader always sees the writer’s latest commit, searched with
cosine-distance nearest-neighbor queries. Everything else — file
sync cursors, per-file embedding status, a full-text index — lives
in SQLite via better-sqlite3. The two have different access
patterns: approximate nearest-neighbor search over high-dimensional
vectors isn’t the same problem as exact keyword lookup and
transactional state, so they don’t share a store just because it
would look simpler on a diagram.
That split is also what makes a newly connected Drive searchable immediately instead of after an overnight indexing run. Cataloging a Drive records every file as pending in seconds — metadata only, no embedding yet. A search hits the SQLite full-text index, which covers pending files too, synchronously embeds whatever’s in the top results if it isn’t embedded yet, and a background drain fills in the rest over time. Hybrid search blends that keyword layer with vector similarity, so the first query against a huge Drive returns something useful instead of a “still indexing” message.
Serving it over MCP
The MCP server is built on the official @modelcontextprotocol/sdk,
registering four tools with zod-typed inputs: search_drive for
semantic search that returns ranked chunks with citation links,
get_file_content for a file’s full reconstructed text,
list_indexed_files for a paginated catalog, and resync to report
embedded-versus-pending coverage. All four run against the same
local pipeline above — there’s nothing an MCP client can ask for
that routes through a server Corpusly doesn’t have.
It’s reachable over two transports. Claude Desktop and Claude Code
spawn it over stdio — one process, one cross-process write lock, so
exactly one running instance embeds and writes while any others
serve read-only. A separate loopback HTTP transport, bound to
127.0.0.1 only and bearer-token authenticated, with Host and
Origin validation checked before auth even runs, lets a background
daemon serve the same tools continuously — deliberately with no
TLS, since encrypting a loopback socket doesn’t defend against
anything a same-host attacker doesn’t already have. A small proxy
transport lets a stdio-speaking client transparently ride that same
HTTP session, so the spawned process and the long-running daemon end
up talking to one server, not two. (Building on an existing
Drive-MCP server was the other option; none of them exposed an
incremental change feed, which a continuously-synced index needs, so
this one talks to the Drive API directly instead.)
The constraint that shapes everything downstream
MCP’s stdio transport uses stdout as the literal wire protocol:
every JSON-RPC message is a line of stdout, so one stray
console.log anywhere in the dependency tree corrupts the stream
and silently drops the connection. Trusting every dependency,
including transitive ones pulled in later, to never write to stdout
isn’t a bet worth taking. A small shim demotes noisy libraries'
console calls to a stderr logger structurally — pdf.js, in
particular, floods console.warn during parsing — so the constraint
is enforced once, centrally, instead of by convention everywhere.
The background daemon that makes all this continuous is held together by two separate locks, one for “at most one daemon,” one for “at most one writer,” plus a priority mutex so an interactive search never waits behind a bulk re-embedding pass. None of that is visible from the MCP client’s side. It’s the same principle as the transport proxy: however a client is configured to reach Corpusly, the answer underneath is always exactly one thing doing the writing.
Local-first as a compliance property, not just a pitch
Indexing an entire Drive needs Google’s restricted drive.readonly
scope; the narrower drive.file scope is per-file and doesn’t
cascade to folder contents, so it doesn’t work for this at all.
Restricted scopes normally mean Google’s own costly third-party
security assessment — but that assessment is triggered by
server-side handling of the data. Corpusly qualifies for Google’s
local-client exemption instead, because the local-first architecture
isn’t a claim, it’s the literal absence of a server to assess.
Privacy-first turned out to also be the cheaper compliance path — a
nice property to have arrived at by taking the architecture
seriously, rather than the other way around.
Closer
None of this reads as one privacy feature; it’s what falls out of refusing to add a server in the first place. Every step from a Drive API call to a searchable vector runs on the machine that asked for it, and the one place a hosted embedding provider exists in the code is a stub that isn’t implemented yet, not a fallback quietly wired up behind a config flag. Corpusly’s in early access, with a waitlist live at corpusly.ai.