How we benchmark Oynix

This is the companion to our benchmarks page. The page shows what the numbers are. This shows how they were produced: the harness, the judging protocol, how every figure was recorded, how competitors were configured and what they spent, and the controls that make the comparison fair.

We measure three things. Conversational memory, because that is what Oynix is. Code retrieval, because a memory layer that cannot find the right file is not useful. Indexing wall clock, because a developer decides whether to keep a tool in the first two minutes.

One result frames everything below. Oynix reaches the top of the memory table with retrieval that needs no API key and costs nothing per query, while every other memory system in that table spends model tokens to build its memory. The free arm wins.

Every number here was produced by running the product, not a copy of it. Retrieval goes through the same entry point an AI agent calls over MCP, and ingest goes through the product's own path.

How a run is actually executed

A run is not a script that calls an API and prints an average. It is a sequence with a checkpoint at every stage, because the failure modes that matter are the silent ones.

1. 1

   1. Preregister the sample

   The seed is fixed and a checksum of the selected question IDs is written to disk before any scoring code runs. Nothing about which questions count can change after we see a result. For memory the sample is 600 LoCoMo questions at seed 20260817.

2. 2

   2. Provision an isolated stack per arm

   One process per arm, one graph per process. This is not tidiness. Two indexing arms once shared a process and the second silently inherited the first one's graph, so arms now run isolated with an assertion that the node count belongs to this arm.

3. 3

   3. Ingest through the product path

   Documents go in the way a user's documents go in, so date stamping, chunking and enrichment are the product's rather than the harness's. After ingest we verify the stored item count. A system that silently accepted a fraction of what we sent is the single most common way a competitor gets an unfairly low score.

4. 4

   4. Retrieve, once, and prove it

   The retrieval function is wrapped in a counter and the harness asserts exactly one call per scored question. The local retrieval branch is confirmed reached on every question rather than sampled. Any outbound HTTP call during a scored run is a hard failure, because the classic bug is a network call that fails quietly and returns an empty list.

5. 5

   5. Read with a pinned model

   Every arm gets the same reader model, the same prompt, the same temperature and the same context budget: five units, 8,000 characters, whole units admitted in rank order. Retrieval is the only variable. A stronger reader on one side would invalidate the entire table.

6. 6

   6. Judge, record, and test

   An LLM judge scores each answer against the reference, every field is written to a per-question record, and comparisons run a paired exact McNemar test over the questions where two arms disagree.

The LLM-as-judge protocol

Most of the trust in a benchmark like this rests on the judge, so the judge gets the same discipline as the systems under test.

The judge receives the question, the reference answer, and the candidate answer. It does not receive the system name, the retrieved context, or any indication of which arm produced the answer. It cannot reward us for being us because it does not know which one we are.

The judge model, prompt and temperature are pinned and identical across every arm in a comparison group. Changing the judge creates a new comparison group rather than silently replacing old rows, because a judge change moves every number at once and a table that mixes them is meaningless.

• The judge is blind to the arm. No system name, no retrieval trace, no ordering signal. Answers from every arm are scored under identical conditions.
• Judge noise is measured, not assumed. We feed two arms identical input and count how often the judge disagrees with itself. On our setup a difference of about two questions in a hundred and fifty is inside that noise. We do not quote a margin smaller than the noise floor we measured.
• Prompt and model are recorded per run. Judge model, version, temperature and prompt hash are written into the run record, so any figure can be traced back to the exact judging configuration that produced it.
• A deterministic evaluator is preferred where one exists. Where a benchmark ships an official evaluator we use it rather than reimplementing scoring. A reimplementation that agrees with itself is not evidence, and two implementations agreeing cannot catch a defect they share.

The judge never sees which system produced an answer. That single property is what lets a comparison between our system and a competitor's mean anything at all.

How the numbers are recorded

Nothing is computed from a log line or a printed average. Every scored question writes a structured record, and the published figure is an aggregation over those records rather than a number someone copied down.

Why the paired test, and what p = 0.0518 means

Two systems answering the same 600 questions do not produce two independent samples. They produce 600 paired observations, and most of those pairs agree. Treating them as independent inflates confidence dramatically, which is how a benchmark post ends up claiming a decisive lead from a margin that would flip on a re-run.

The paired exact McNemar test looks only at discordant pairs: the questions where one system was right and the other wrong. Those are the only observations carrying information about which system is better. Questions both systems answered correctly tell you about the benchmark, not about the difference.

Against mem0 we lead by seventeen questions across 195 disagreements, and the test returns p = 0.0518. That sits directly on the conventional 0.05 threshold. It is a genuine lead, measured on a preregistered sample, with the reader model and retrieval budget held identical on both sides. We report the exact p rather than rounding it in either direction, because the number is more informative than any label we could attach to it.

Against supermemory, BM25 and zep the same test returns p below one in a million. Those margins are not close.

Repeated runs, and why a single reading is never the answer

Every timing-sensitive figure on the benchmarks page comes from repeated execution, not one measurement. This matters more than it sounds, because provider latency moves further in an afternoon than most of the differences anyone would want to advertise.

The indexing configuration comparison is the clearest case. A first sweep suggested smaller batches were 28% faster: 51.0 seconds against 71.0. A repeat of the identical configuration minutes later measured 84.2 seconds, slower than the baseline it had just beaten.

So we interleaved the arms A/B/A/B/A/B, so drift would hit both configurations equally, and ran the ladder repeatedly. Across every run the default configuration spanned 69.3 to 82.4 seconds while the tuned arm spanned 47.9 to 84.4, a 76% range on one configuration. The conclusion is that the defaults are correct and no tuning is warranted, which we only reached by running it five and six times instead of once.

The concurrency ladder was run the same way, at 10, 40 and 100 lanes, and it shows a ceiling: 40 lanes beat 10, and 100 lanes are worse than 40. We publish the ladder rather than the winning row, because a single best reading would imply parallelism scales indefinitely.

• A tight repeat is not a noise floor. Two runs minutes apart share whatever the provider was doing during those minutes. Agreement between them proves the minutes were calm, not that the measurement is stable.
• Interleave or you are measuring the clock. Running arm A five times and then arm B five times measures the difference between two time windows as much as between two configurations.
• No speedup below roughly 2x is treated as a result. That threshold comes from the observed spread, not from taste.

Running it once produced a 28% speedup we would have been happy to publish. Running it six times showed it was not there.

What each competitor was given, and what it spent

Configuration is the difference between a comparison and a smear. Every competitor here runs self-hosted and uncapped, with every model variable pinned by hand, because hosted free tiers cap things in ways that fail silently and always in our favour.

The cost column below is what each memory system spent on model calls to build and search its own memory over the run. The reader model is identical across every arm, so it is excluded: this isolates the memory system itself rather than the answering step everyone shares.

Two arms in the memory table need no API key: ours and the lexical baseline. Ours scores highest of every system tested. Every arm that pays a model to build its memory scores below the one that does not.

What Oynix itself spent, which is nothing

Worth stating plainly, because it is the part most easily lost in a table. Across every lane we ran, Oynix spent nothing on model calls to do its own work.

Retrieval is lexical fused with an on-device ONNX embedder of about 110 MB. It runs on the user's machine, needs no API key, and costs nothing per query. That is the path that produced the top score on the memory table, and it is the same path that leads all four code lanes.

The only model spend anywhere on our side of a comparison is the reader and the judge, and both of those are shared identically by every arm. They are the fairness controls, not our cost. Strip them out and the Oynix column is zero everywhere.

• Memory lanes. $0. No API key involved in building or searching memory. Every other memory system in the table pays a model to construct its memory before it can answer anything.
• Code lanes. $0, and keyless on both sides. Oynix and graphify are the only two arms here that need no key at all.
• Structural indexing. $0. Parsing, the code graph and the search index are derived from the source itself, with no network and no model.
• Reader and judge. Shared identically by every arm, ours and theirs. Pinned to the same model, the same prompt and the same budget, which is precisely what makes the comparison mean anything.

The one place Oynix does spend, and why it is optional

There is exactly one lane where Oynix spends model tokens: optional indexing enrichment. A model writes a natural-language description for each function and an embedder turns it into a vector. That is what the with-a-model indexing figures measure.

It is worth being clear about what this is, because it is not an Oynix-specific tax. Any AST-based code tool that wants docstring-level understanding and cosine similarity between documents has to do the same work. graphify and the other parser-based tools face the identical cost if they want that capability. The difference is that in Oynix it is opt-in: leave the key unset and you get the full structural graph, lexical search and impact traversal for nothing.

It is also worth being clear that the benchmark numbers on our page do not depend on it. The memory and code retrieval results were produced by the keyless path. Enrichment buys semantic retrieval on top, and on real repositories we have measured better numbers with it enabled, but nothing published here needs it.

Structural indexing is free and always has been. Enrichment is the one paid step, it is optional, and every AST-based tool that wants the same capability pays the same kind of cost.

The ledger, and why the metered figure is a floor rather than a total

The per-system costs above are metered per lane, and they are accurate for what they measure: what each memory system spent on model calls inside that run. They are not the programme total, and it is worth being precise about why.

Our harness ledger meters our own calls. It cannot meter what a third-party binary decides to spend on our key, and on one occasion that gap was large. A competitor's binary resolved its own much stronger model rather than the one we configured and spent roughly 91.7 million tokens in a day. The harness ledger read about $8 for that period. The provider dashboard read about $45. Roughly $33 of real spend was invisible to the meter that was supposed to be watching it.

So we treat every metered figure as a lower bound and say so. Across the whole programme, run over many sessions and many arms, real spend is materially higher than any single lane's meter suggests. We publish the per-lane metered numbers because they are the ones that are actually comparable between systems, and we publish the dashboard delta alongside them because the meter alone has already been proven wrong once.

Every model variable is now pinned explicitly, and a vendor binary gets inspected before an environment variable is trusted. Indexing spend is estimated rather than metered, at roughly ten cents for a 659-description run.

• Metered per lane. What each system spent inside a specific scored run. Comparable between arms, and the basis of the cost column on the benchmarks page.
• Observed on the provider dashboard. The only figure that catches spend the harness cannot see. On the worst day it was about $45 against a metered $8.
• Programme total. Higher than any single metered figure, across many sessions and many arms. We do not quote a precise programme total, because a number we cannot reconstruct from records is not a number we should publish.

A ledger that meters only your own calls will under-report by exactly the amount that matters most: what someone else's code spent on your key.

The controls that make a comparison fair

These are the things a reader should check before believing any benchmark, ours included.

• Same reader model on every arm. Reader, judge and extraction all pinned to the same model across a comparison group. Competitor models are set explicitly rather than left to a default, because defaults change without telling you.
• Same retrieval budget on every arm. Five units, 8,000 characters, whole units admitted in rank order. A system given a larger window is not being compared, it is being flattered.
• Same questions, scored pairwise. Every arm sees the identical preregistered sample, which is what makes the paired test valid.
• Stored-item verification before scoring. We confirm what a competitor actually retained before we score it. This has repeatedly turned a flattering result into an honest one.
• Failures counted, not dropped. A timeout or a process failure stays in the denominator. Removing it would quietly inflate every rate.
• The product path, not a harness path. Retrieval goes through the entry point an agent calls. A benchmark that measures an internal function proves you can write a benchmark.

Scope: what we measure and what we do not claim

A benchmark page is defined as much by its scope as by its results.

• LongMemEval is measured and not quoted. It is reader bound to saturation: the gold evidence reaches the context window in 147 of 150 cases and the reader still misses roughly 47%. Retrieval is effectively solved there, so a strong score would describe the reader model rather than us. Quoting it would be claiming someone else's work.
• No SWE-bench result is claimed. Three indexing repositories live in a directory named after SWE-bench and we used only the source trees. None of its tasks, patches, harness or scoring is involved in any figure we publish.
• Indexing wall clock has no published benchmark. No paper defines it and no standard tool measures it. We wrote the harness, it lives in the repository, and it states exactly what it counts. Timings come from a monotonic clock wrapped around the engine's own log statements, so phase boundaries are the engine's rather than ours.
• One machine, one model, one embedder for indexing. Every indexing figure comes from a single hardware profile, stated rather than implied.

Datasets and papers

Every link below was checked rather than recalled. Where a benchmark ships an official evaluator we use it rather than writing our own.

• The memory dataset. Every memory number we publish is scored on it

  Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024.

  arXiv:2402.17753Project page

• The paired significance test behind every comparison

  McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 1947.

• The lexical baseline in every retrieval lane, and the hardest arm to beat on code

  Robertson and Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 2009.

• The on-device embedder in our keyless retrieval lane. No API key, zero per-query cost

  Xiao, Liu, Zhang, Muennighoff. C-Pack: Packed Resources for General Chinese Embeddings. Model used: bge-base-en-v1.5, ONNX.

  arXiv:2309.07597Hugging Face

• Measured and deliberately not quoted, because it is reader bound

  Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025.

  arXiv:2410.10813OpenReviewGitHub

• Source trees only. No result claimed or implied

  Jimenez, Yang, Wettig, Yao, Pei, Press, Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.

  arXiv:2310.06770GitHub

Competitor papers

We benchmarked these systems ourselves, self-hosted and uncapped. Their own published numbers use different setups, models and samples, so they are not comparable to our tables and we do not reproduce them here. The papers are linked so anyone can check what each system claims for itself.

• The closest competitor in the memory lane

  Chhikara, Khant, Aryan, Singh, Yadav. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. 2025.

  arXiv:2504.19413GitHub

• Run as open-source Graphiti on FalkorDB

  Rasmussen et al. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. 2025.

  arXiv:2501.13956

Repositories and tooling

The four indexing repositories are real open source projects, used only as source trees.

• Timing. Python's monotonic clock wrapped around the engine's own log statements. No profiler and no benchmarking framework were used for any indexing figure.
• Rate limits. Read from the provider's own response headers rather than assumed, so a throttled run is visible as throttling rather than as slowness.
• Tests. Run under pytest. The indexer suite passes 3,349 tests with zero failures.

What we are measuring next

This page grows with the product. Every capability we ship gets a lane, and a lane only appears here once it has a preregistered sample, a pinned configuration and a paired test. Nothing goes up as a placeholder score.

These are the suites in progress. They are listed so the scope of what is measured today is unambiguous, and so it is obvious what is not yet claimed.

1. 1

   The same lanes at n = 10,000

   The next target is a preregistered sample roughly sixteen times the current one. It is the only way to settle the comparisons that sit close today: a margin at the threshold on 600 questions resolves one way or the other on 10,000. Every competitor's model spend scales with the sample and lands around twenty times what it is now. Ours stays exactly where it is, because keyless retrieval multiplied by any sample size is still zero. That asymmetry is also the practical reason we can afford to run it at all.

2. 2

   SWE-bench, as an agent-outcome study

   Today we use SWE-bench source trees only and claim no result from it. The version worth running is different: identical agent scaffold and identical model on both sides, with Oynix as the single variable, measuring whether an agent resolves more issues when it has the graph than when it does not. We have run early work in this direction and it is not currently a win, which is exactly why it belongs on a roadmap rather than in a results table.

3. 3

   Ingestion integrity

   Exact graph assertions rather than node counts: node and edge precision and recall per edge type, source-location accuracy, and idempotency across a repeated index. A correct graph has to be proven before retrieval or impact quality means anything.

4. 4

   Impact and blast radius, in the product

   The file-retrieval benchmark already published is a different mechanism from the product's traversal. The traversal gets its own suite with hand-labelled callers by hop depth, cross-repository seams, and negative cases where similarly named endpoints must not be linked.

5. 5

   MCP interoperability

   Whether real agents can discover, select, call and recover from Oynix tools across different clients. A correct graph is not useful if a client picks the wrong tool or mishandles a structured result. Every acceptance case makes a real tool call; listing the tools is a wiring preflight, not evidence.

6. 6

   Organizational memory over real sources

   The memory lane today is conversational. The product's actual job is memory over tickets, docs, threads and code history together, with timestamps respected so no answer uses information that did not exist yet.

7. 7

   Local performance profiles

   Cold and warm indexing, incremental re-index after a one-file change, and query latency at p50, p95 and p99, reported per hardware profile rather than as one machine's number.

8. 8

   Code understanding, not just retrieval

   Retrieval measures whether the right evidence arrives. The next question is whether the answer built from it is complete, grounded and correctly cited, scored against atomic facts rather than overall impression.

A suite appears on the benchmarks page when it has a preregistered sample, a pinned configuration and a paired test. Until then it lives on this list.

Why this rigour is the product claim

Oynix exists because AI coding agents start every conversation from zero. They read code well and know nothing about the organization that wrote it. Memory is the fix, and memory is only worth something if retrieval works when an agent asks.

That is why the figure we lead with is the one measured through the shipped path rather than through a harness. A benchmark that measures an internal function proves you can write a benchmark. A benchmark that goes through the same entry point an agent uses proves the product works.

It is also why the cost column sits next to the accuracy column. Reaching the top of a memory benchmark is one thing. Reaching it with retrieval that needs no API key, runs on the user's own machine and costs nothing per query is a different claim, and it is the one that survives contact with a real engineering budget.

Documentation explains what happened. Memory explains why. Nothing is lost.

If you find a defect in how we measured any of this, we want the report more than we want the number. The method is the product claim.