Benchmarks · last run 17 August 2026
Measure what the graph actually changes.
Memory, code retrieval and indexing, each measured against self-hosted competitors on the same model, the same retrieval budget and the same preregistered questions. Every comparison carries its paired significance test.
On LoCoMo at n = 600, measured through the shipped path, Oynix answers 422 of 600 correctly. That is the highest score of every system we tested, ahead of supermemory, BM25 and zep by margins that clear significance outright, and ahead of mem0 with the paired test sitting right on the conventional threshold at p = 0.0518.
And it does it for $0. Every other memory system in this table spends model tokens to build its memory. Oynix and BM25 are the only two that spend nothing, and the free one wins. In the code lanes the same holds: Oynix and graphify are the only keyless arms, and Oynix leads every lane.
How we measured all of this →- Every arm in a comparison runs the same model for reader, judge and extraction. Competitor models are pinned explicitly, never left to a default.
- Comparisons use paired exact McNemar on the questions where two arms disagree. We report the number of disagreeing pairs alongside the p value, because a margin over 600 questions means nothing without it.
- Samples are preregistered: seed fixed and a checksum written before any scoring.
- Anything we could not verify is in the limits section rather than left out.
Memory
This is the product, not a harness. The run goes through the same search an AI agent calls over MCP and ingests through the product's own path, so what is scored here is exactly what a user gets. Zero questions produced an empty context, and the retrieval that produced this score needs no API key.
Conversational memory
LoCoMo, preregistered n = 600. Accuracy, 0 to 75%. Every arm runs gpt-4o-mini.
Seed 20260817, checksum written before scoring. k = 5, 8,000 characters, whole units admitted in rank order. Memory cost is what each system spent on model calls to build and search its own memory over the run. The reader model is identical across every arm, so it is excluded to isolate the memory system itself.
| System | Accuracy | Correct | Memory cost | Paired vs Oynix |
|---|---|---|---|---|
| Oynix | 70.33% | 422/600 | $0 (keyless) | reference |
| mem0 | 67.50% | 405/600 | $0.197 | 195 discordant, p = 0.0518 |
| supermemory | 55.50% | 333/600 | ~$0.50 | p = < 0.000001 |
| BM25 baseline | 55.00% | 330/600 | $0 (keyless) | p = < 0.000001 |
| zep (Graphiti) | 39.67% | 238/600 | $0.78 | p = < 0.000001 |
How far this moved
- What the product shipped that morning (0.5567)+88 questions, p < 0.0001
- Our best experimental arm (0.7100)60 vs 64 discordant, p = 0.7877
- mem0 (0.6750)+17 questions, p = 0.0518
The middle row is the one worth reading twice. Our best experimental arm needed an extra answer-verification pass, and we had assumed that put a hard ceiling on anything actually shippable. The shipped path matches that arm without it. The expensive step turned out to be unnecessary, so the number at the top of this page is the one users already have rather than one waiting on a feature.
Where the remaining headroom is
Fact-extraction systems still score higher on adversarial questions, the ones whose answer is not in the conversation at all, because absence is easy to check in a list of facts. We reach the top of the table without paying an extraction model to get there, and that category is where the next gain comes from. Everywhere else we lead, and we lead from a retrieval path that costs nothing to run.
How the fixes were chosen
One diagnostic selected both. Two deficits with opposite causes, and therefore disjoint remedies.
An aggregate score cannot tell a retrieval problem from a reader problem
Coverage on the x axis, accuracy-when-covered on the y axis. Both 0 to 100%.
- Retrieval bound
- Reader bound
- Healthy
- Weak both sides
- multi-hopRight almost every time the evidence arrives. It usually does not.
- adversarialThe evidence is there and the answer is still wrong.
- single-hopHealthy on both axes.
- temporalHealthy after the dated-chunk fix.
- open-domainWeak on both sides, n = 25.
| Category | Gold fully in window | Accuracy when it was | Diagnosis |
|---|---|---|---|
| multi-hop | 26% | 86% | Retrieval bound |
| adversarial | 88% | 39% | Reader bound |
| single-hop | 85% | 90% | Healthy |
| temporal | 77% | 80% | Healthy |
| open-domain | 48% | 33% | Weak both sides |
Dated chunks
+0.0700 (p = 0.000039)
temporal 0.2549 → 0.6569
Keyless fusion
+0.0466
multi-hop 0.4000 → 0.5250, single-hop 0.7656 → 0.8608
Two candidate fixes were killed by measurement before being built. A retrieval confidence gate cannot work, because adversarial questions have the highest median top score of any category, 13.25, above single-hop's 13.03. A global abstention instruction is a wash: +17 adversarial, −14 elsewhere, p = 0.7493.
Code
Four retrieval lanes against the closest competitor and against the lexical baseline that keeps everyone honest. Oynix and graphify are the only two arms here that need no API key, both run at zero cost, and both get the same retrieval budget. Oynix leads every lane.
Code retrieval
Accuracy, 0 to 100%. Both sides keyless, both zero cost, same retrieval budget.
- Oynix
- graphify
- BM25
issue → coden = 300
issue → code @5n = 300
doc → code, time disjointn = 794
impact / blast radiusn = 290
| Benchmark | n | Oynix | graphify | BM25 |
|---|---|---|---|---|
| issue → code | 300 | 88.33% | 80.33% | 87.33% |
| issue → code @5 | 300 | 69.67% | 36.67% | 61.33% |
| doc → code, time disjoint | 794 | 88.41% | not run | 87.78% |
| impact / blast radius | 290 | 81.03% | 47.24% | 73.10% |
- BM25 also beats graphify on issue → code, 0.8733 against 0.8033. Our win is real and BM25's win is also real, and quoting one without the other is misleading.
- Two different things in our own documentation are called impact. The 0.8103 above is a file-retrieval benchmark. The product's impact traversal is a different mechanism and it measured zero lift over a no-tools baseline. These must never be conflated.
Indexing
Wall clock to index a repository. There are two modes, they do different work, and they belong to different orders of magnitude. Mixing them into one number would describe neither.
Without a model configured
No AI calls. No network. No spend.
Oynix parses the repository, builds the structural code graph, and writes the search index. Everything is derived from the source itself: files, symbols, imports, calls, inheritance, ownership. This is what you get with no API key set, and it is what the search index is built on. It finishes in seconds on every repository we have measured, and it costs nothing because nothing leaves the machine.
- The full structural code graph
- Lexical search across the whole repository
- Impact and caller traversal
- Zero cost, zero network, works offline
With a model configured
Everything above, plus a written and embedded description for every function.
The same structural pass runs first, then a model writes a natural-language description of every function and an embedder turns each one into a vector. That is what makes semantic retrieval possible: an agent can ask what something does rather than what it is called. It is the slow part by a wide margin, and it is the only part that costs money.
- Everything in the structural mode
- A natural-language description per function
- Semantic retrieval, not just lexical matching
- Roughly ten cents and about a minute for a 659-function repository
Mode one: no model, no network, no spend
Structural indexing does not scale with node count
Wall clock in seconds against graph nodes. No model, no network, no spend.
- pocketbase/js-sdk
- psf/requests
- sphinx-doc/sphinx
- pylint-dev/pylint
Timings come from Python's monotonic clock wrapped around the engine's own log statements, so phase boundaries are the engine's and not ours.
| Repository | Files | Nodes | Parse | Graph write | Search index | Total |
|---|---|---|---|---|---|---|
| pocketbase/js-sdk | 54 | 1,055 | 1.88s | 0.7s | 0.05s | 2.66s |
| psf/requests | 125 | 3,812 | 15.82s | 0.81s | 0.1s | 16.76s |
| sphinx-doc/sphinx | 616 | 17,725 | 8.64s | 2.06s | 0.52s | 11.31s |
| pylint-dev/pylint | 1,944 | 24,358 | 5.85s | 1.67s | 0.43s | 8.08s |
Mode two: with a model configured
Measured with gpt-4o-mini plus text-embedding-3-small at 1,536 dimensions. The second row is what linear scaling in function count looks like: roughly seventeen times the descriptions, roughly seventeen times the wall clock. Structural indexing stays in seconds regardless.
| Repository | Nodes | Descriptions | Description phase | Embed phase | Total |
|---|---|---|---|---|---|
| pocketbase/js-sdk | 1,055 | 659 | 59.6s | 4.45s | 70.35s |
| sphinx-doc/sphinxcaveat | 17,725 | 11,023 | 996s | 74.5s | 1,100s |
This is the only place Oynix spends anything, and it is optional. Any AST-based code tool that wants docstring-level understanding and cosine similarity between documents has to do the same work, graphify included. Leave the key unset and you get the full structural graph, lexical search and impact traversal for nothing. None of the memory or code retrieval numbers on this page depend on it.
sphinx-doc/sphinx: This figure carries an unresolved caveat in our source notes. It is a single reading, and provider latency on a run this long swings wider than on a short one. Treat it as the order of magnitude, not the number.
Where the 70.35s goes
pocketbase/js-sdk, 1,055 nodes, 659 descriptions, gpt-4o-mini plus text-embedding-3-small at 1,536 dimensions.
- Parse0.9s1.3%
- Graph write0.47s0.7%
- Batch planning1.96s2.8%
- Description generation59.61s84.7%
- Embedding, all 659 in one pass4.45s6.3%
- Graph write of descriptions and vectors0.67s1.0%
- Vector index build0.43s0.6%
- Search index1.53s2.2%
Description generation is 84.7% of the cost and it scales linearly with function count. That is the honest limit of the current design.
| Phase | Time | Share |
|---|---|---|
| Parse | 0.9s | 1.3% |
| Graph write | 0.47s | 0.7% |
| Batch planning | 1.96s | 2.8% |
| Description generation | 59.61s | 84.7% |
| Embedding, all 659 in one pass | 4.45s | 6.3% |
| Graph write of descriptions and vectors | 0.67s | 1.0% |
| Vector index build | 0.43s | 0.6% |
| Search index | 1.53s | 2.2% |
Tuning does not beat the defaults, and we checked properly
A first sweep suggested smaller batches were 28% faster, 51.0s against 71.0s. A repeat of the same configuration minutes later measured 84.2s. Interleaving the two arms A/B/A/B/A/B so drift hits both equally, the tuned arm won 1 pair of 3. Across every run the defaults spanned 69.3s to 82.4s and the tuned arm spanned 47.9s to 84.4s, a 76% spread on one configuration. The run-to-run variance is larger than any difference between settings, so 70.35s at defaults is the number, and no tuning is recommended.
The tuned arm is bimodal: it lands either near 48–51s or near 82–84s, never between. That is a real signal about something switching, and it is unexplained. It is not a speedup we can claim.
Concurrency has a ceiling
Measured on the same repository. More parallelism stops helping and starts hurting. We publish the regression rather than the best row.
- 10 lanes201.48s
- 40 lanes142.23s
- 100 lanes160.21s
What we measure next
This page grows with the product. Every capability we ship gets a lane, and a lane appears here only once it has a preregistered sample, a pinned configuration and a paired test. Nothing goes up as a placeholder score.
What n = 10,000 costs each system
Projected from the current sample of 600, not measured. Every system here pays a model per item it ingests and per query it answers, so cost tracks sample size almost linearly and lands at about 20x today's figure. One row is not an estimate.
| System | At n = 600 | Projected at n = 10,000 |
|---|---|---|
| Oynix | $0 | $0 |
| mem0 | $0.197 | about $4 |
| supermemory | ~$0.50 | about $10 |
| zep | $0.78 | about $16 |
| BM25 baseline | $0 | $0 |
This is the practical reason keyless matters. Scaling a benchmark sixteen times multiplies everyone else's bill and leaves ours untouched, and the same arithmetic applies to a team running the thing in production rather than in a harness.
Every method, dataset and significance test behind these numbers is written up in full, including the harness defects we found in our own favour and the results we voided.
Read the methodology →