Skip to main content

Results, including the inconvenient parts

The primary metric is MRR@10 on the prefix-v2 search-query task. all uses a transductive closed candidate vocabulary; train contains train-observed targets only.

Trace configPoolCoverageResetStateful adapterPrior + residualSupport proxy
Kimiall1.00000.33400.37890.45590.4132
Kimitrain0.76830.29790.33000.45420.3775
GLM-5.1all1.00000.32670.36550.53860.5522
GLM-5.1train0.79170.29330.32410.53700.5273

The stateful adapter improves over reset in all four rows. The explicit prior terms are larger than the state-only gain, which is why the project does not present the full improvement as “memory.”

Leakage and conversion audit​

The canonical conversion contains 57,142 next-argument samples with zero temporal-order violations. All 212,620 stored history entries preserve observed tool responses. These checks establish causal availability; they do not establish semantic generalization.

Tool-heldout matrix​

Across 2,676 BGE-M3 heldout samples and nine tools:

  • maintained state versus reset: +0.0159 MRR@10, session-clustered p=0.0001;
  • support proxy versus maintained state: -0.0032 mean delta;
  • three tools lose under support gating.

On the stricter train-only heldout audit, coverage is 23.54%, state improves by +0.0055 (p=0.0031), and the support delta is +0.0020 but non-significant (p=0.2904).

No-loss support claim rejected

Support gating can help aggregate in-distribution ranking while hurting cold heldout tools. Report the worst-tool losses and confidence tests, not only mean MRR.

URL negative control​

Maintained state underperforms reset on the Kimi URL task. URL arguments are strongly affected by literal repetition and normalization, so URL results are treated as a negative control rather than clean state-memory evidence.

External TAU boundary​

Three session-split TAU-bench seeds preserve temporal validity and response coupling, but train-only target coverage averages 11.92%. Ranking improvements cannot retrieve a target absent from the candidate vocabulary. Candidate generation is therefore the binding external bottleneck.

Rewrite-model refresh​

MiniMax-M3 and GLM-5.2 rewrites were evaluated as query-rewrite baselines, separate from the state-adapter claim. Their metadata, paired comparisons, and provider settings are tracked in the rewrite matrix artifacts; they do not replace the frozen-encoder core result.

Machine-readable sources​

Read the evidence contract before citing a result outside its evaluated boundary.