Ansys AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Ansys AI AI

Opus 4.8 vs Opus 4.7 vs GPT 5.5 on n=50 real tasks from 2 open source repos

Opus 4.8 is finally out - how good is it actually? In this benchmark, I compared Opus 4.8 vs the rest of the frontier (GPT 5.5, Opus 4.7, Composer 2.5) on n=50 real tasks from 2 open source repos (graphql-go-tools and sqlparser-rs, Go and Rust respectively) representing complex backend software engineering work across a variety of tasks. The important part is that these repos are arbitrary - I could have tested the models on my repo, using my tasks, to see how well the frontier performs on domain-specific tasks. The goal of this is to explore, with granularity, how a benchmark like this is constructed and what it can tell us about model behavior. Let's go! Disclosure up front: I build Stet, the local eval tool I used to run this Full post with expanded detail and dataviz available here: https://www.stet.sh/blog/opus-48-vs-gpt-55-vs-opus-47-vs-composer-25 TL;DR The king is back - Opus 4.8 is the craft leader in both Go and Rust, and dominates the two premium-reasoning arms (GPT-5.5 high, Opus 4.7 xhigh) on the cost-quality plane - equal-or-better craft while cheaper + leaner. Only loss is raw price: Composer 2.5 is ~6.5× cheaper on Rust (and ~7× on Go) but materially weaker on craft. cost vs custom score How strong is each claim: the craft win over Composer is decision-grade in both repos, and over GPT-5.5 on Rust; the Go craft edge and the exact ordering among the "premium" models are only directional (n=25, one grader pass). "Decision-grade" vs "directional" is defined in the stats note below. Why I ran this Most public benchmarks answer binary task-outcome questions - did the model satisfy the grading condition set out by the task author. This is helpful for measuring model intelligence, but is notably different from how real engineers use models. As a SWE in an enterprise codebase, I don't care just about whether Opus 4.8 passes the tests. I want it to write idiomatic, maintainable code that doesn't introduce subtle bugs. It needs to write high-quality diffs that would get approved and merged by my teammates. Attempting to answer the question of "should I move my team from Opus 4.7 to 4.8 / from Claude to GPT-5.5 / try Composer to cut cost?" is almost impossible to answer from public data alone - you need hands-on, anecdotal experience using the models on your own code (or local benchmark data) to understand performance in reality. I'm not claiming this is universal benchmark - it's one run, two repos, n=25 each. Methodology Each task is real merged PR/commit from the source repo. The agent is dropped into a Docker container with a frozen repo snapshot, a prompt to do the task, and one attempt. We then apply the patch + runs the task's tests in an isolated container. This is then graded beyond test pass/fail: Equivalence (same behavioral change as the human patch?) Code review (would a reviewer accept it?) Footprint risk (extra code touched vs human patch) Craft/discipline (8 graders: clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, diff minimality). One run per task, single seed; judge = GPT-5.4, blinded to which model produced the patch with manual spot-checks. There's no human calibration pass, so trust direction of deltas over absolute scores. Details: Models = Opus 4.8 (high, Claude Code); Opus 4.7 (xhigh, Claude Code); GPT-5.5 (high, Codex); Composer 2.5 (Cursor) One integrity note: this corpus isn't network-sandboxed, so I audited for contamination. One Composer Rust result turned out to be a gold-leak (the agent fetched the merged PR) which I caught, swapped for a clean rerun, and which only widened Opus's lead once removed. A broader set of tasks (Composer and Opus alike) touched the network in ways I judged benign and kept as valid. As an aside, I've also been using these evaluations as an "autoresearch" optimization loop, not just a benchmark. I tell my agent something like "make AGENTS.md better for this repo"; it proposes an edit, runs Stet on historical tasks, figures out where the candidate was better / worse and why, and iterates to improve the evaluation numbers. Comparisons How to read the numbers below. With n=25 per repo, no single grader is conclusive - the smallest craft gap one grader can reliably catch (~0.34–0.49 on the 0–4 scale) is bigger than most real gaps here. The signal is agreement. Think coin flips: one landing heads tells you nothing, but flip 10 and get all heads and something's up. When 8–11 independent graders all lean the same way, a sign test on that consensus is significant even when no single grader is. I tag a result decision-grade (DG) when it survives multiplicity correction (BH-FDR), and directional when it's consistent but doesn't clear that bar. vs GPT-5.5 high - better craft, leaner everywhere, and cheaper in Rust (Go cost lands ~par). Opus writes better code in both repos. Craft-mean leads on Rust (3.28 vs 2.94, DG - 4 graders survive) and on Go (2.90 vs 2.72), though G

Love Claude auto-fill giving itself praise

100% misread it the first time as “both look good, keep it up” submitted by /u/OsbornHunter [link] [comments]