Six weeks ago I read seven code-graph and agent-memory tools at source level, decided none of them could be adopted where a security reviewer signs off first, and started building my own. I called it the shadow and promised that the next post would say how it turned out. It has a name now: kivuli, which is Swahili for shadow.

Since 4 September it has grown to 269 commits on main, 35 decision records and eight signed releases, the last of them this weekend. That’s enough history to measure it against the tools that defined it, and measuring is most of what I’ve been doing; it’s also where the interesting results are, including two I’d rather not have found.

One thing first. This post is about outcomes, not mechanisms. I’ll say what kivuli does and what the numbers were, and I’ll name the ideas it borrowed from other people’s public code, but where something is my own I’m describing what it achieves rather than how. If you were hoping for a build guide, this isn’t one.

What it is, in a paragraph

kivuli is one Go binary that lives on a developer’s machine and never talks to the network. It keeps a per-repository graph of the code’s structure (files, symbols, imports, calls, inheritance) without storing a single line of source, so every answer is a path and a line range and the agent opens the file itself. Beside that sits a per-user memory of what coding agents did and decided, built from the transcripts Claude Code, Copilot and opencode already write to disk, with secrets taken out before anything is stored. An agent reaches both through one MCP tool with twelve modes, and every question it asks goes into a hash-chained audit log that names who asked, what they asked and what came back. Eight languages plus Markdown; macOS and Linux.

That last sentence about the audit log is the whole reason it exists. Everything else on that list, some tool in the previous post already did.

Against the seven

The table compares kivuli with each tool on the axes that decided the build. The figures for the other seven are from my source reviews on 4 and 7 September and will have moved since; kivuli’s are from v0.2.3.

  MCP surface Default egress Redaction at ingest Audit names who asked Signed releases
openwiki 6 tools PostHog, model provider Prompt instruction No npm provenance
claude-mem 14 tools PostHog, hosted gateway Manual <private> tag No None
deja-vu 1 tool Version check in doctor Yes, with counters No query text cosign, SBOM
code-review-graph 30 tools One model download None No Static PyPI token
codegraph 1 shown, 8 total Telemetry, daily update check None No Provenance
graphify 10 tools Model auto-detect, CDNs Exclusion by filename Opt-in, no identity PyPI OIDC
ripwire Several, some write files None None No .sha256 sidecar
kivuli 1 tool, 12 modes None, by construction Yes, no off switch Yes, hash-chained SSH-signed, SBOM, SLSA provenance

A table flattens the interesting part, so here is each one in turn, with the number that moved.

codegraph

The tool I use every day, and the one whose “No data leaves your machine” failed on the first grep. kivuli’s equivalent claim is held by four separate checks rather than a sentence: a test that fails if any package imports a network library, a test-only guard that fails if anything reaches the default HTTP client or resolver, a check over the linked binary’s symbols for an HTTP client or a TLS handshake, and a committed inventory of which dependencies import network packages, diffed on every build. Off by default and absent by construction are different properties, and I wanted the second one to be provable by somebody who doesn’t trust me.

From codegraph I took the staleness banner, almost unchanged. Every answer names the files that have changed since indexing instead of pretending they haven’t.

code-review-graph

Thirty MCP tools, which its own documentation puts at about 8,000 tokens of description on every turn. kivuli’s single tool description is capped at 600 tokens by a test and currently costs about 520, roughly a fifteenth; the detail of each mode lives in per-mode skills that cost nothing until somebody opens one. That cap has been the most annoying constraint in the project and probably the most valuable, because every new mode has to argue for its sentence.

Its weighted impact walk is the idea I borrowed. Its “~65x token reduction” is the claim I deliberately didn’t copy, and I’ll come back to why under the numbers.

graphify

The best parser design in the set, a generic tree-sitter walker driven by a per-language table, and kivuli’s parser follows it. Where kivuli differs is dependencies: graphify carried 29 runtime packages and code-review-graph 45 transitive, against kivuli’s five direct dependencies under a CI-enforced cap of six. Its document pipeline needs a model. kivuli reads Markdown deterministically instead, headings as symbols and mentions of code as edges, which costs some intelligence and buys an answer that’s the same bytes every time.

openwiki

openwiki needs a model to do anything, and kivuli has no model client at all, so on the face of it there’s nothing to compare. In practice openwiki ended up as the most useful repository in the whole project, because it was the first real TypeScript codebase I pointed kivuli at. That run found 862 of its 1,232 relative imports unresolved, because modern ESM imports name the .js file the compiler will emit rather than the .ts file that exists; fixing it took the inferred call edges from 3,301 to 4,373 and halved the heuristic ones. Its relocatable evidence hashes, a SHA-256 over every symbol’s source span, are in kivuli’s schema.

claude-mem

Fourteen tools, a separate vector database and PostHog on by default, with secrets kept out only when the user remembers to wrap them in a <private> tag. kivuli’s redaction runs in the one function that writes memory records and there’s no environment variable, flag or policy key to turn it off; a test walks the tree for the names someone would reach for if they tried to add one. The last ingest on my macOS Developer Laptop read 26 transcript sources and 7,618 records in 1.98 seconds and removed 194 secrets on the way in. I didn’t know there were 194 secrets in my own transcripts. That number alone justified the memory half.

deja-vu

The closest relative, and the one I learnt most from: redaction with counters, recall at the point of action, and a written statement of its own limits. On dependencies it still beats kivuli outright (it has none), and I’m not going to pretend otherwise.

What kivuli adds on the memory side is the part a reviewer asks about after redaction. You can forget a session permanently, and it stays forgotten when its transcript is read again. You can export everything held about you in one file. Retention is enforced, but the window stays unset until the operator picks a number, because a tool that deletes somebody’s memory on a figure it invented is worse than one that keeps it and says the control is off. deja-vu, by contrast, quietly kept transcripts past Claude Code’s own 30-day cleanup.

ripwire

The seventh tool, which independently reached most of my design and shipped it. kivuli took three things from it: task-seeded ranking, so an agent can ask in prose instead of naming a symbol; the disclosure vocabulary, so every count says whether it’s a floor; and ceilings on pathological input in front of the C grammars. The ceilings got tested properly: eleven fuzz targets, one hour each under the address sanitiser, 142.7 million executions and no finding.

The differences are on the other side of the trust boundary. ripwire’s MCP tools can write files anywhere on the machine and one option runs an arbitrary shell command. kivuli’s tool can write exactly one thing, a note of at most 4,000 bytes into the user’s own memory store, redacted like everything else. Fifty prompt-injection payloads planted in comments, signatures, file names and transcripts go through every mode, every CLI renderer and every hook on every build, and none of them may come out able to act on whatever reads it.

The numbers, and the rule I held them to

Every number in the previous post’s “claims that didn’t survive” section had the same flaw: whoever ran it chose the baseline, the sample and the pass mark, often after seeing the result. So before running kivuli’s gates I committed a document fixing the repository, the commit, the machine and the pass mark for each, on 23 September; the commit is the timestamp. A choice changed after it gets recorded under it with the date and the reason. None has been.

Speed on a real monorepo

microsoft/vscode at tag 1.139.0, picked in advance as the biggest and most interconnected TypeScript I could find: 13,571 files, 264,281 symbols, 1.57 million edges. Medians of three runs on my macOS Developer Laptop, an Apple M4.

  Measured Budget
Cold index 63.2s 10 minutes
Warm sync after one file changed 7.38s 10 seconds
Query p50 105µs  
Query p95 25.0ms  
Peak memory 1.80GB  

Both pass. The cold index is a tenth of its budget. The warm sync is the number I watch, because its cost follows the size of the repository rather than the size of the change, and a tree about a third larger than vscode would miss the ten seconds; I’ve said so in the write-up rather than claim a bigger monorepo than I’ve run. The parser limits refused one file, a 4 MiB colouriser fixture, and the answers disclose it.

Does impact analysis actually predict anything?

Instead of a token-reduction ratio against a baseline nobody pays, I measured whether kivuli predicts the files a developer actually had to change. For each merged multi-file change, build the graph at the parent commit, hold one changed file out, ask kivuli what the rest of the change affects, and check whether the held-out file appears. Four repositories fixed beforehand (kivuli, Prometheus, FastAPI, Excalidraw), separate calibration and test sets of twenty merges each, and a pass mark of a 30% median with no repository under 15%.

It passed: median 38.5%, lowest 19% on FastAPI.

Then I found the problem with my own pass mark. On three of the four repositories the floor kivuli chose for itself went low, and at Excalidraw’s a single prediction ran to 938 symbols, a sixth of the codebase. That finds co-changed files by volume and tells a reviewer very little. At a size a person would actually read, the median is 31% and FastAPI drops to 6%; a lot of FastAPI’s history is sweeps across hundreds of documentation examples that change together because a policy changed, which no call graph can see. The gate passes by its letter, I didn’t fail it after the event, and the next registration bounds the prediction size up front.

How does that compare with the other tools’ retrieval figures? Honestly, it doesn’t. ripwire’s 58.3% is strict file@10 on 60 Python questions; deja-vu’s 85.3% is session-level hit@1 on memory recall. They measure different questions, so putting them in a row would be exactly the kind of number this section exists to avoid. The difference I’ll stand behind is procedural: kivuli’s figure was registered before it was run, the repositories weren’t mine to pick afterwards, and the finding that weakens it is published beside it.

The gate it failed

v0.2.0 added Markdown, and I reran every gate against it the same week. The benchmark held within 2%. The holdout, run exactly as registered, fell to 5% on FastAPI, under the minimum.

Nothing had got worse at predicting code. Indexing documents changed which merges qualified for the sample, so half the test sets on two repositories became documentation-only changes, and the graph almost never predicts a changelog entry. Rebuilding the original v0.1.0 test merges and scoring them with both binaries showed v0.2.0 finding every co-changed code file v0.1.0 found, and a few more. The same run also turned up a real bug: function calls could resolve to Markdown headings, and in FastAPI 734 call edges pointed at a heading in an agent skill file because every tutorial calls FastAPI(). Fixed that afternoon. The record still says v0.2.0 failed as registered, because it did.

What it did for its own development

The point of pre-registering was to keep me honest. What I hadn’t expected was how many defects the measuring instruments would find in kivuli itself, almost all of them invisible to the 530-odd unit tests.

Running the impact study on kivuli’s own history found that the scoring tool understated recall, by requiring a reviewer to type a fully qualified name. It also found that a call to f.Sync() on a plain file handle was being resolved to kivuli’s own index.Sync, which dragged precision on that change down to 15%. The resolver matches names, not types, and I knew that in the abstract. What I didn’t have was the count.

Feeding in a compiler-grade index from scip-go gave me the count. Of the 6,515 name-resolved edges in kivuli’s own graph, the type checker confirmed 4,516 and called 1,999 wrong, nearly a third; 1,503 of those were a repository function sharing a name with a library method the code was really calling, time.Now() landing on struct fields called Now and sql.Row.Scan on three unrelated Scan functions. kivuli can now take that index and correct its own graph with it, and the shape of every answer changed when it did.

The benchmark found two more. The generated sample I’d been timing against turned out to be far tidier than real code, at one point 1.6 edges a file against kivuli’s 13.9, and once I made it as dense as the real thing, a one-file sync on 5,000 files took 4.7 seconds and re-resolved 960 of 1,200 files; after the fix it re-resolves 14. Then SQLite’s query planner started choosing the wrong index on a seed lookup, so a callers query took 2.8ms where it should take 41µs, and no test noticed because a test asks whether the answer is right, not how long it took.

The threat model, written on 24 September, found that v0.1.0 followed symbolic links out of the repository, a real vulnerability in a released binary; v0.1.1 fixed it with a changelog entry telling anyone who’d indexed an untrusted repository to rebuild. The same day it found that a repository’s own git configuration could make git run a program when kivuli called it, and that a committed policy file could inflate one answer from 5 KB to 71 KB. Fuzzing the transcript parsers, which I’d forgotten to do, found invalid UTF-8 getting into the memory store within the first second.

Behind all of that sit rules that turned out to be load-bearing. No file over 2,000 lines. Five direct dependencies. An architecture document that a test holds to the package tree, so it can’t drift the way ripwire’s had. And a compliance mapping against NIST SSDF, SLSA and two OWASP standards, where every claim points at a file or a test and the build fails if the evidence stops existing. Last post I said the answer to one person and an agent producing a maintenance liability was constraints that actually bite. So far they bite. kivuli’s busiest day was 58 commits; ripwire’s was 133.

The part that doesn’t flatter it

kivuli has a substitution meter, which reports what share of an agent’s navigation calls went to kivuli rather than to reading and grepping files. It’s the number that says whether the tool is being used, and I have two sets of it.

On my macOS Developer Laptop it’s close to nothing: two answers metered, both from 23 and 24 September. The Claude Code transcripts for kivuli’s own repository since then say the same thing louder, with the agent calling the kivuli tool four times against 238 Read calls and 1,773 Bash. Most of kivuli was written before there was a graph worth asking, which explains some of that and not all of it.

My Ubuntu Developer Workstation is more interesting. kivuli has been installed there, hooks and all, since 28 September, and across three weeks and eleven working days its meter says 78% of navigation went to kivuli. I’d love to quote that figure. It’s wrong.

The meter counted any audit entry from kivuli as an answer, and that included the hooks. A hook isn’t the agent choosing kivuli; it’s kivuli speaking up on its own before a shell command or an edit. Of the 1,950 answers, 1,945 were hook injections. The agent itself asked the MCP tool five questions in three weeks (two notes to remember, a recall, an explain and one callers), and against the 562 grep-shaped shell calls and file reads the meter caught in the same window, that’s under 1%.

So the two halves landed very differently. The memory side worked the way deja-vu suggested it would: 1,643 times before a shell command and 288 times before an edit, kivuli put something from an earlier session in front of the agent without anybody asking, a median of 722 bytes each. Whether those injections changed what the agent did is a separate question, and the meter can’t answer it. The graph side, the half with all the benchmarks and the holdout, the agent barely touched. My guess is habit; grep and Read are what it reaches for in every other repository, and a 520-token description is one voice among many in its context.

The same report turned up something worse. kivuli audit verify on the Ubuntu Developer Workstation says the chain is broken at line 81, and nobody edited the file. Each hook runs as its own process, the log only serialises writers inside one process, and when two hooks fire at the same moment both read the same end of the chain and write the same link. Forty concurrent queries reproduce it on demand: forty entries with eight sequence numbers between them, 37 of them claiming the same predecessor as some other entry. The chain still shows that something is wrong. What it can no longer do is tell tampering apart from two hooks firing together, and for a log whose whole job is to be believed by somebody else, that’s close to the same failure. It’s fixed now, though not yet in a release: every append takes a lock that holds across processes and catches up with whatever the others wrote before it chains, and a test runs eight real processes against one log to keep it that way. Writing that test found a second bug, a running writer that kept appending to the log after the runbook’s recovery had moved it into the evidence folder. The meter is fixed too: it now counts the questions an agent chose to ask and prints the hooks on a line of their own, outside the rate, so a figure like 78% can’t come out of it again. It had also been counting my own replays of old questions as the agent reaching for kivuli, which nobody would have guessed from the number.

So the honest answer to how kivuli improved its own development is that it did so as an instrument and a set of rules, not yet as the thing the agent reaches for. The benchmark, the holdout, the overlay, the fuzzing and the replay all found real defects, and the constraints kept the codebase legible at a pace that would otherwise have buried it. Then the first usage report from my Ubuntu Developer Workstation found two more, both in the controls I built the tool around rather than in the graph.

That’s what the four-week gate is for, and the Ubuntu Developer Workstation’s three weeks show what it should measure: questions the agent chose to ask, counted separately from what the hooks volunteered, with the pass mark written down before the first day. The 200-record redaction review of my own transcripts is still open too, and so is the warm sync that a monorepo a third larger than vscode would break.

If you take one thing from this: write the pass mark down before you run the test, then publish the finding that weakens your own result next to the result. It cost me a clean headline on the holdout and a failed gate on v0.2.0. It also found a vulnerability, a third of my call graph being wrong, an audit log that breaks under its own hooks and an agent that ignores the graph it was given, and I’d rather know all of it.