Orchard Was Quietly Revised on 30 July, and Its Benchmark Numbers Moved

Orchard, the open-source agentic modelling framework from Microsoft Research, is not new: the paper was submitted on 14 May 2026. What is new is that it was last revised on 30 July 2026, and the revision moved the headline numbers. The blog post and most of the coverage still quote the May results. If you are evaluating agent frameworks on published benchmarks, you are probably comparing against figures that have been superseded.

What Orchard actually is

The problem Orchard sets out to address is a practical one, and anyone who has tried to reproduce an agent paper will recognise it. Building state-of-the-art agentic systems generally requires proprietary infrastructure: custom sandboxes, closed training pipelines, and datasets that most teams can neither access nor reproduce. The result is a field where published results are hard to verify and harder to build on.

At the centre of the framework sits Orchard Env, described as a thin, Kubernetes-native environment service. Rather than shipping an opinionated agent, it exposes generic primitives: sandbox lifecycle, command execution, file I/O, network policy, a REST API, and lightweight agent injection. That list is worth reading slowly, because it is essentially the infrastructure layer every team rebuilds privately when they start running agents at any scale.

On top of that base, three specialised domains ship with it: Orchard-SWE for software engineering, Orchard-GUI for browser navigation, and Orchard-Claw for personal-assistant tasks.

The numbers that changed

This is the part worth being precise about, because two different sets of figures are circulating and both are real.

The May version reported 64.3% on SWE-bench Verified after supervised fine-tuning, and 67.5% after SFT plus reinforcement learning. Those are the numbers in the Microsoft Research blog post and in nearly every article written about Orchard since.

The 30 July revision reports 69.7% with RPR-based reinforcement learning, and 73.0% with value-model reranking, on a Qwen3.5-35B-A3B backbone. That is a jump of more than five points on the same benchmark, from the same team, on a paper carrying the same arXiv identifier.

The other two domains are documented with equal precision in the revision: 68.4% average success across WebVoyager, Online-Mind2Web and DeepShop for the GUI agent, and 59.6% pass@3 on Claw-Eval, rising to 73.9% when paired with the stronger ZeroClaw. The training budgets are small enough to be worth noting: 0.4K distilled trajectories and 2.2K open-ended tasks for the GUI work, 0.2K synthetic tasks for Claw.

Why a silent revision matters more than it looks

A revised preprint does not make headlines. There is no press release for a v2, no changelog entry in most feeds, and the URL does not change. The paper simply gets better while the internet keeps quoting the version it read three months ago.

For anyone selecting a framework on the strength of published benchmarks, that is a real trap, and it cuts both ways: you may be under-rating a framework that has improved, or comparing two systems whose numbers were measured under different conditions and different dates. The practical habit is unglamorous and effective: check the submission history on arXiv before quoting a benchmark, not just the abstract.

What we take from it

Two things, stated plainly. First, the interesting contribution here is not the score but Orchard Env: an open, Kubernetes-native substrate for running agents in isolation is the piece most teams are missing, and it is the piece that is usually rebuilt badly in-house. Second, benchmark numbers on fast-moving preprints have a shelf life measured in weeks.

If you are building agents that touch real systems, the sandbox question comes before the model question. That is the order in which these problems bite.

Sources

Orchard: An Open-Source Agentic Modeling Framework, arXiv 2605.15040, submitted 14 May 2026, last revised 30 July 2026. Microsoft Research blog, Orchard: An open framework for scalable agentic AI.