https://agentdiagonstics.com/ Newly added papers, benchmarks, and tools on AI agent diagnostics and optimization, curated by CausaMind. en Mon, 26 Oct 2026 12:00:00 GMT https://arxiv.org/abs/2610.28601 https://agentdiagonstics.com/resources/evolving-agent-interfaces Mon, 26 Oct 2026 12:00:00 GMT Optimization Computer-use agents are capped by hand-built interfaces and hand-built knowledge of when to use each. Agent S4 treats interface design as an agent capability: a meta-agent examines task agents' trajectories on synthetic training tasks without success labels, and builds and revises interfaces and usage knowledge in a shared repository, while population-based evolution selects repositories by performance on validation tasks the meta-agent never sees. With Qwen3.8-27B and the evolved repository, Agent S4 reaches 21.3% on OSWorld V2 — 3.8 times the same model with fixed interfaces and above GPT-5.5 and Claude Opus 4.8 at a fraction of their cost — and lifts ScienceBoard success from 41.4% to 58.6%. Results are reported on two benchmarks, and the meta-agent's own compute cost is not part of the headline comparison. https://arxiv.org/abs/2609.40169 https://agentdiagonstics.com/resources/scholarevolve Mon, 26 Oct 2026 12:00:00 GMT Optimization An agent harness — the software governing tool use, memory management, and task execution — can be improved while the underlying model stays fixed, but existing automated methods react to observed failures using only the meta-agent's prior knowledge. ScholarEvolve instead draws on the research literature: it organizes harness evolution directions into functional modules, uses topic modeling to identify distinct improvement strategies per module, implements them, and evaluates their combinations, with new publications incorporated over time. On AppWorld Challenge it raises Qwen3.5-27B goal completion from 49.6% to 63.6%, and on Tau2-Bench Telecom it raises GPT-5.4-mini pass@1 from 72.7% to 81.9%. Gains are demonstrated on those two benchmarks, not across agent harnesses in general. https://arxiv.org/abs/2610.31066 https://agentdiagonstics.com/resources/dual-exam Mon, 26 Oct 2026 12:00:00 GMT Evaluation Multimodal agents and world models are usually evaluated together, so a failure cannot be blamed on action selection or on state evolution. Dual Exam separates the two: agents are tested under fixed environment response mechanisms across physical, digital, social, and scientific domains, and world models under fixed action interventions. The agent track compares 10 configurations on 1,088 items in 15 reporting categories, and the world model track covers 400 cases across eight scenarios. A 150-case pilot of one coupled pairing records zero task successes, which motivates component-level diagnosis — though the authors state that the value of these separate scores for predicting coupled success or sustained reciprocal learning remains to be established. https://arxiv.org/abs/2610.31130 https://agentdiagonstics.com/resources/post-audit-mirage Mon, 26 Oct 2026 12:00:00 GMT Failure analysis Offline checks are supposed to tell whether an agent update stays safe after deployment, but they only see verifier-visible evidence. The authors construct matched safe and harmful agent lifecycles that produce identical verifier-visible evidence while an isolated scorer records opposite deployment truth, and prove an error-abstention tradeoff for this boundary. Their executable benchmark spans authorization, scheduling, and shared-capacity settings, and their IdentifiedRangeMonitor returns deploy, hold, cannot determine, or unsupported from fresh group-labelled harm outcomes. Monitoring recovers the missing distinction in favorable settings, but stays undecided for rare groups, small effects, or missing observations — so a verifier's verdict is only as good as the evidence it can actually see. https://arxiv.org/abs/2610.30284 https://agentdiagonstics.com/resources/swe-verifier Sun, 25 Oct 2026 12:00:00 GMT Optimization Coding agents waste computation on poor action choices, but sampling and verifying alternatives at every step adds substantial overhead. SWE-Verifier is a lightweight generative model trained from Qwen3.5-4B that scores a proposed action to decide when candidate sampling is needed, then ranks the resulting actions before execution. Using the mini-swe-agent harness it improves resolution from 75.47% to 77.20% on MiniMax-M2.7 at 1.09x cost and generalizes to DeepSeek-V4-Flash without training on its actions, while verifying fewer than 4% of steps — matching or beating all-step verification at substantially lower cost. Gains are measured on the mini-swe-agent setting and two models, not across agent harnesses in general. https://arxiv.org/abs/2610.30573 https://agentdiagonstics.com/resources/prefdiag-bench Sun, 25 Oct 2026 12:00:00 GMT Failure analysis Personalized GUI agents must let user preferences guide sequential execution over changing GUI states, but existing benchmarks mostly report overall task success, hiding how preferences should alter strategy and when they should take effect. PrefDiag-Bench categorizes preferences into five types and adds a stage-wise diagnostic framework with 1,664 probes tracing where failures occur; across eleven GUI agent systems, performance on preference-personalized tasks is substantially lower than on non-personalized ones, with the best model reaching only 47.17% on Process preferences under explicit profiles. The two bottlenecks are selecting relevant evidence from user history and recognizing when a preference should take effect; explicitly providing relevant evidence improves end-to-end success by up to 7.06%. The benchmark covers mobile GUI tasks and the studied preference types, not personalization in general. https://arxiv.org/abs/2610.31376 https://agentdiagonstics.com/resources/certified-execution-harness Sun, 25 Oct 2026 12:00:00 GMT Agent harness Safe tool execution for LLM agents under prompt injection is formalized with Petri nets: the harness between model and tools becomes a supervisory control problem in which marks represent capabilities, approvals, and budgets, so an injected request whose preconditions are unmet or already spent is refused regardless of its wording. Safety transfers to the running system under four stated assumptions, and the paper characterizes the maximally permissive least-intrusive supervisor for a replanning loop in which each retry costs an inference call — showing the conventional supervisor can be arbitrarily more intrusive than a stricter safe one. On an autonomous release agent, synthesis shows safe deployment requires a step the supervisor can refuse after a rollout starts; the guarantees are only as strong as the four assumptions and the modeled harness. https://arxiv.org/abs/2610.30738 https://agentdiagonstics.com/resources/agenttriage Sun, 25 Oct 2026 12:00:00 GMT Failure analysis When tool-use agents write files, run code, or send data, their final responses do not explain the resulting security events, and heterogeneous runtimes expose evidence through incompatible interfaces. AgentTriage is an executable contract for replayable security diagnosis: runtime overlays normalize native observations, a deterministic pipeline joins invocation and result lifecycles into a typed provenance graph, and conclusions must cite canonical runtime events or abstain when mandatory evidence is missing. Given the same normalized trace, overlay version, and policy oracle, a diagnosis can be recomputed and re-checked without language-model judgment; across five runtimes it reaches 0.858 grounded joint diagnosis on 120 adjudicated cases, with a 0.036 unsupported-claim rate. Results are bounded by the runtimes and cases studied, and by what the overlays can observe. https://arxiv.org/abs/2610.30491 https://agentdiagonstics.com/resources/trace-vs-label Sun, 25 Oct 2026 12:00:00 GMT Evaluation Agent evaluation often reduces execution traces to labels — final-answer correctness, tool verification, fault propagation — but those labels are produced by parsers, detectors, and harness rules that may not capture what the trace shows. In a tool-use setting built on GSM8K with an occasionally corrupted calculator, the authors audit three trace-level detectors across six production models and find systematic disagreement: a key failure mode is silent tool-output override, where the model uses a different, often correct value without re-calling the tool, so a verification detector records no re-check. Human review of 90 traces agrees with automated labels on final correctness and fault impact but disagrees on verification in 14 of 81 cases, and step and token budgets shift one model's score from 21/30 to 27/30 — so labels should ship with the trace evidence behind them. https://arxiv.org/abs/2610.31483 https://agentdiagonstics.com/resources/transferable-tool-selection Sat, 24 Oct 2026 12:00:00 GMT Optimization As agents operate over corpora of more than ten thousand tools, selecting the right tool for a complex multi-step instruction is a central bottleneck: descriptions do not fit in context, and relevance retrievers rank without knowing what is actually needed. The paper trains a small open model with reinforcement learning to perform the decomposition-and-selection role previously driven by proprietary LLMs; on ToolBench, a post-trained Qwen3-8B outperforms Llama-4-Maverick (402B), and the trained model transfers directly to different retrievers and new tool corpora in different domains. The evidence is ToolBench-style selection accuracy, not end-to-end task completion. https://arxiv.org/abs/2610.31461 https://agentdiagonstics.com/resources/adaptive-skill-granularity Sat, 24 Oct 2026 12:00:00 GMT Agent harness Skills let agents package procedures into modular units, but execution poses a runtime control problem: the same task may suit primitive tool calls, a macro skill, or a hybrid that starts macro and falls back to micro recovery. Adaptive Skill Granularity treats granularity as an action-abstraction policy conditioned on task structure, execution state, and macro-skill failures, combining a feature-based granularity controller with a finite-state recovery executor. Under a deterministic coding-agent protocol with verifier-defined success, runtime routing and explicit macro-to-micro recovery beat fixed macro execution on robustness and recover controlled macro failures that fixed-granularity execution leaves unresolved; fixed macros remain efficient when they apply. https://arxiv.org/abs/2610.31276 https://agentdiagonstics.com/resources/ripple Sat, 24 Oct 2026 12:00:00 GMT Optimization Long-horizon coding agents can observe a failure, write a plausible critique, and still continue down the same incorrect trajectory — the Critique-to-Action Handoff Gap. RIPPLE internalizes process-level recovery guidance into a single self-recovering policy: a prefix-only teacher supplies evidence-grounded recovery supervision without access to future actions or terminal outcomes, then reinforcement learning assigns stage-specific credit to reflection triggering, diagnosis, and corrective action, with local validity signals gating trajectory advantages. Across four repository-level software-engineering benchmarks it improves the base policy by a mean 4.8 points with no auxiliary critic at deployment; gains concentrate where the prefix holds enough diagnostic evidence and the repository state stays reversible. https://arxiv.org/abs/2610.30792 https://agentdiagonstics.com/resources/trace2world Sat, 24 Oct 2026 12:00:00 GMT Agent harness Scaling GUI reinforcement learning needs environments that preserve real-app behavior, but inferring behavior and code together turns partial traces into mocks with wrong state transitions. Trace2World separates global behavior reconstruction from implementation: it reconciles traces into a shared, source-linked app model, compiles an executable environment, and uses independent replay of source workflows to localize mismatches for bounded repair. From 1,570 traces it builds 50 environments; on 200 workflows across ten apps, 91.1% reproduce end to end versus 59.8% for the strongest task-local baseline, and source-grounded validation retains 94.0% of simulated training gains on real devices versus 54.7% under self-validation. https://arxiv.org/abs/2610.31593 https://agentdiagonstics.com/resources/simulated-control-evals Sat, 24 Oct 2026 12:00:00 GMT Evaluation Control evaluations test whether safeguards stop a model that is intentionally trying to cause undesired outcomes, but control settings may not represent deployment environments and tasks. Simulated control evaluations replace the environment with a language model that simulates tool responses; on LinuxArena, SCEs predicted safety with 3.6 percentage-point RMSE across five monitors at a 5% audit budget, versus 10.1 for a separate BashArena control setting, and reproduced attack-strategy and monitor-reasoning distributions with over tenfold lower KL divergence than BashArena. Strategies discovered through simulation also improved monitor performance on held-out LinuxArena attacks, approximately matching gains from executed strategies — though validation so far is confined to LinuxArena-style terminal tasks. https://arxiv.org/abs/2610.29869 https://agentdiagonstics.com/resources/compaction-laundering Fri, 23 Oct 2026 12:00:00 GMT Failure analysis Coding agents that run long enough must compact their context, and this step is an exploitable trust boundary: harnesses that flatten the typed conversation into a single plain string — marking each message with a text label like [User]: — destroy message provenance in code before the summarizer runs. An attacker who controls anything the agent reads can forge a user turn that survives compaction as a genuine instruction: the attack succeeds 87–100% of the time across five flattening harnesses (pi, opencode, Cline, Aider, OpenHands), up to ~908,000-token context windows, and 0% of the time across four role-typed harnesses (Codex, Gemini CLI, Qwen Code, Continue). A laundered standing approval drove real tool-running agents to execute attacker-controlled curl | sh and to steer plans away from attacker-designated files; a code-level provenance check catches 77% of laundering, and escaping role labels at the serializer fixes the defect. https://arxiv.org/abs/2610.31366 https://agentdiagonstics.com/resources/afterdeletebench Fri, 23 Oct 2026 12:00:00 GMT Observability Deleting an original memory record need not remove information carried by its descendants: summaries and derived state can retain deleted content. AfterDeleteBench is a controlled behavioral audit that compares exposed-then-deleted agents against matched never-exposed controls, so post-deletion recovery is attributable to the original exposure. A single query recovers the deleted secret from Qwen2.5-7B-Instruct in roughly two thirds of cases, and exact reconstruction from separately stored components under a public composition rule succeeds 62–99% of the time across Qwen, Mistral, and Phi-3 — no surviving verbatim copy required. Provenance-aware closure eliminates observed recovery, while an exact-literal write filter only blocks reinsertion: deletion must be evaluated behaviorally at the agent-memory-system level, not as a storage operation. https://arxiv.org/abs/2610.31650 https://agentdiagonstics.com/resources/episem-bench Fri, 23 Oct 2026 12:00:00 GMT Evaluation A tool-using agent can carry out a prohibited action when a tool response omits a safety-relevant fact — even when the user request is benign and every returned statement is true. EpiSEM-Bench is an executable benchmark of 300 matched unsafe/safe task pairs across six domains that systematically suppresses critical facts to measure unsafe execution under evidence omission, and determines the minimum observed failure cardinality across repeated trials. An independent audit tool checks whether agents actively seek missing facts and withhold prohibited actions while completing permissible ones; across ten models, resistance to missing evidence varies widely and does not necessarily translate into useful recovery. https://arxiv.org/abs/2610.28361 https://agentdiagonstics.com/resources/falca Fri, 23 Oct 2026 12:00:00 GMT Optimization Long-horizon deep search breaks down when early mistakes compound: agents anchor on an incorrect candidate and collect partially supporting evidence. FALCA frames deep search as a propose-falsify loop — a Searcher proposes candidate answers or sub-answers while independent Falsifiers actively seek counterevidence, check unmet constraints, and produce falsification reports for research-state revision. Falsification-aware training combines SFT warmup on falsification trajectories with agentic RL over parallel rollouts and hierarchical credit assignment across research stages, state transitions, and actions; on five search benchmarks it beats backbone and harness ablations while reducing confirmation bias and error accumulation. https://arxiv.org/abs/2610.28201 https://agentdiagonstics.com/resources/mad-rsi Fri, 23 Oct 2026 12:00:00 GMT Agent harness Existing recursive self-improvement methods optimize a harness for a single LLM, coupling the learned harness to that model and requiring costly re-optimization for each new model. MaD-RSI treats models as data: it optimizes one shared harness across multiple heterogeneous LLMs by folding their diverse failure trajectories into training, so model differences become optimization evidence rather than only a post-hoc generalization check. The paper also introduces two complementary metrics for assessing cross-model harness optimization effectiveness; experiments across multiple datasets and LLMs show better generalization over existing approaches at lower training cost. https://github.com/vercel-labs/durable-harness https://agentdiagonstics.com/resources/durable-harness Thu, 22 Oct 2026 12:00:00 GMT Agent harness Vercel's open-source adapters that make existing coding-agent harnesses — Pi, OpenCode, Claude Code, Codex, fx — durable by running each session as a workflow on the Workflow SDK's durable backend. Every step of a turn (intent before, outcome after each model and tool call) is written to a durable log, so a crashed function resumes the turn on a new one without repeating finished calls; each tool declares a replay policy, with non-idempotent tools like refunds never re-run. Session files (Claude Code transcripts, Codex rollouts, OpenCode event stores) are mirrored into the log, and the same agent runs on Vercel, a local machine, or a self-hosted Postgres World.