SWE-agent

SWE-agent AI

SWE-agent AI

SWE-agent AI

SWE-agent AI

SWE-agent AI

SWE-agent AI

SWE-agent AI

SWE-agent AI

SWE-agent AI

SWE-agent AI

Tested 4 brand new frontier models (2 Chinese, 1 diffusion, 1 agent-focused) with a riddle that has no logical shortcut. One of them fabricated sources four times in a row.

I've been running the same weird test on every new model that ships: a riddle that can't be solved by pattern-matching or web search, only by actually connecting two unrelated things. This time I added a second riddle and ran both against four models that all shipped in the last few weeks: MiMo-V2.5-Pro (Xiaomi), MiniMax M3, Mercury 2 (Inception Labs, diffusion-based), and LongCat-2.0 (Meituan). Rules: no web search, no context given beforehand, up to 3 hints only if requested, same prompt copy-pasted for all four. Riddle 1: What connects an elegant lady walking a small dog to the most famous character played by actor Walter Koenig? (Koenig played Chekov in Star Trek. The surname is a nod to Anton Chekhov, who wrote "The Lady with the Dog.") Riddle 2: What connects actor Henry Winkler to Microsoft? (Winkler played Fonzie in Happy Days. Fonzie cameos in Weezer's "Buddy Holly" video, directed by Spike Jonze. That video was bundled on the Windows 95 install CD as a multimedia demo.) Riddle 2 has zero logical path to it. You either have that exact chain sitting in your weights or you don't. Good test for what a model does when it simply doesn't know. Results, riddle 1: MiMo-V2.5-Pro: solved cold, zero hints. Even correctly identified the dog breed in the actual short story (Pomeranian) without being asked. MiniMax M3: solved cold, zero hints, with genuinely fun reasoning shown along the way. Mercury 2: needed 1 hint, clean reasoning once it had it. LongCat-2.0: needed 2 hints. But here's the thing. LongCat on riddle 1, before any hints, with web search off: it told me, confidently, with fake citation markers, that Walter Koenig's wife was known in Star Trek fan circles for walking a small Pekingese at conventions. None of that exists. Total fabrication. I gave it the hint that the answer is in the character's surname, expecting a correction. Instead it decided "Chekov" sounds like "Chihuahua," then went right back to the fabricated wife story and repeated it even after I told it that was wrong. Only got there after hint 2 basically spelled out the answer. Riddle 2, nobody solved cold. Mercury 2 needed both hints, got there clean. MiniMax needed both hints, and threw out some entertaining guesses on the way (its first theory: Henry Winkler and Bill Gates share the hidden name "Henry," since Gates' full name is William Henry Gates III — a real fact, wrong riddle, and it said so itself instead of presenting it as the answer). LongCat again did the fabrication thing, worse this time. Before asking for a hint: claimed Winkler voiced a 1976 Sega arcade game called "Fonz." Made up. After hint 1, it threw out three different music videos as candidate answers back to back: a Kanye West video that isn't Spike Jonze, a will.i.am video that also isn't Spike Jonze (acknowledged mid-sentence, offered anyway), then Fatboy Slim's "Praise You" (real Jonze video, explicitly stated to have nothing to do with Happy Days, offered as the answer anyway). Four fabrications across two riddles, several self-contradicting in real time. One honesty note on my own favorite here: MiniMax, while explaining riddle 2, threw in an unprompted detail that the Windows 95 CD also included a bonus video by "the Beastie Boys." Checked it. There was a bonus track, "Good Times," but it's Edie Brickell & New Bohemians, not Beastie Boys. Wrong artist attached to a real fact. Smaller and different in kind from LongCat's stuff (no fake certainty, no repeated insistence), but worth flagging so this doesn't read as "China bad, everyone else perfect." Why I think this actually matters: LongCat beats MiMo on SWE-bench Pro (59.5 vs ~57) and even edges out GPT-5.5 on that metric. It's also trained end-to-end on domestic Huawei silicon with zero Nvidia in the loop, which is a legitimately big deal given export controls. Strong coder, real engineering flex. And it's also the one model here that will hand you a fabricated, confidently-worded answer instead of saying "I don't know," and won't back off when corrected. If you're evaluating any of these for RAG or agentic pipelines, that's the actual risk profile, not the SWE-bench number. Sovereignty over chips and sovereignty over truth are two completely different problems. LongCat solved one and faceplanted on the other. Curious if anyone else has run something similar on these four, or has a nastier riddle to suggest for round 3. https://preview.redd.it/rqyzq7z140bh1.png?width=1536&format=png&auto=webp&s=c0e8435ad0d265aa466f6afcc56ae7e8ec61972b submitted by /u/wikisailor [link] [comments]

I built an AI assistant you can text directly through iMessage/SMS

Full-time SWE here. I’ve been working on a side project with a small team and wanted to share it here because I think people in this community might actually find it useful. It’s an AI assistant that you text directly through iMessage/SMS, basically like texting a friend. The difference is that behind the scenes, it can activate a private AWS server for each user, giving the agent access to its own computer. So it’s not just answering questions. It can actually work through tasks for you. It can help build apps, create websites, make presentations, research things, organize plans, compare options, set reminders, and handle more involved projects on the go. One of my friends used it to plan an entire vacation and get update notifications on places he was interested in staying. That was the moment where it really clicked for me. The value isn’t just “AI in a chat.” It’s being able to text something casually and have it actually start doing the work. It’s free right now while we’re testing, so I made a link for anyone here who wants to try it: textlux.com/awake Would genuinely love feedback from people who test it. Before dismissing it, try giving it a real task. submitted by /u/ryh98 [link] [comments]

DeepSWE: new benchmark looking at how well today's frontier models can actually write code [R]

DeepSWE delivers four advances over existing public benchmarks: Contamination free: Tasks are written from scratch, not adapted from existing commits or PRs, so no model has seen the solution during pretraining. High diversity: Tasks span a broad pool of 91 repositories across 5 languages. Real-world complexity: Prompts are ~half the length of SWE-bench Pro's, yet solutions require 5.5x more code and ~2x more output tokens. Reliable verification: Verifiers are hand-written to test software behavior rather than implementation details. The result is a benchmark that reflects how today's frontier coding agents actually perform in software engineering work. https://preview.redd.it/lacvagyr159h1.png?width=1373&format=png&auto=webp&s=6514340a15d51d7f03da733f08fb3f6a302cac75 It's open-source: https://github.com/datacurve-ai/deep-swe submitted by /u/we_are_mammals [link] [comments]

Some new updates to Papers with Code [P]

Hi folks, Niels here from the open-source team at Hugging Face. I continue working on a revival of paperswithcode.co as we're back to the "age of research" per Ilya Sutskever! Hence, it's important to discover each other's research and build on each other's work, so we can collectively build the next Transformer. Below, I'll go over each of the new features that were recently added. ## Support for SOTA badges Yes, that's right, totally like the old website. You can see that GLM-5.2, for instance, is obviously the hottest blog post today, achieves SOTA on PostTrainBench, and performs well on many other benchmarks. It is displayed whenever a paper gets a score within the top 3 of a given benchmark. Note that these are displayed on any paper feed, including https://paperswithcode.co/tasks/video-classification, for example. https://preview.redd.it/wawma8paeu8h1.png?width=2418&format=png&auto=webp&s=0ba3b6a0eaef231b7f3ca468cc3db4120f1b9e4d ## New trending score The papers are now ranked based on a new trending metric. This is a combination of the GitHub star velocity and the trending score of the linked Hugging Face artifacts (models, datasets, and Spaces). Previously, this only took into account GitHub star velocity. Thanks to this, papers like IndexCache are now trending, which is a core technique behind the trending GLM-5.2 model. https://preview.redd.it/b6g04w2ogu8h1.png?width=2380&format=png&auto=webp&s=13d59bbadd5f8e8295deac2ee6e1e0e3dbc0f40f ## Support for external evals Second, I've added support for "external" evals. This is a feature the legacy PwC website didn't actually have. Oftentimes, a paper has way more evals than the ones introduced in the paper itself. You can now view these third-party evals. Some examples: FrontierSWE and PostTrainBench numbers for GLM-5.2: https://paperswithcode.co/paper/98456#results?task=agents Artificial Analysis has numbers on CritPt, a though physics benchmark. See e.g. https://paperswithcode.co/paper/85629#results?task=reasoning https://preview.redd.it/mfnfdzxpeu8h1.png?width=1914&format=png&auto=webp&s=2b909ecf7c6e3fc088fd0a46fbc56f6859dfaf17 ## More tasks, benchmarks and evals I'm adding more benchmarks and adding evals of more papers. This happens gradually, based on the legacy PwC data available on the hub. Some new benchmarks include: - ImageNet - 10% of the data https://preview.redd.it/wr55g27ofu8h1.png?width=2880&format=png&auto=webp&s=e6e5ef7e3a36cd5aa6d2841b149194239f4ad1e0 - 3D semantic segmentation: https://preview.redd.it/zxgobrnqfu8h1.png?width=2880&format=png&auto=webp&s=6ee2935981825d5d7825709294ddb84a4b7a3ac9 - object counting: https://preview.redd.it/uhv4wbrsfu8h1.png?width=2880&format=png&auto=webp&s=183decb144d9779e41bf12ca58fbaab66cd29cbf and a lot more. Browse all of them at https://paperswithcode.co/tasks ## New domain Papers with Code is now also available from paperswithco.de :) Let me know what is missing, bug/feature requests, and whether you want to contribute! Kind regards, Niels submitted by /u/NielsRogge [link] [comments]

This week in AI: Meta reportedly closing Llama, Anthropic's new model pulled by export controls within a week, and Apple partners with Google for Siri

A few stories from the past week that, taken together, point to a real shift at the model layer rather than just incremental releases: Meta and Llama. Multiple reports indicate Meta is stepping back from open-source Llama in favor of a proprietary program (internally referred to as "Muse Spark," with a new "Avocado" model) under Meta Superintelligence Labs. Llama crossed 650M+ downloads and was arguably the anchor of the open-weights ecosystem, so a pivot to closed development would be significant for anyone relying on that lineage. Anthropic and export controls. Anthropic launched Claude Fable 5 on June 9 (Mythos-class, 1M-token context, always-on adaptive reasoning, notable security/vuln-finding capabilities). On June 12, a US export-control directive reportedly forced Anthropic to suspend access to Fable 5 and Mythos 5. Regardless of the specifics, it's a concrete example of frontier model availability being governed by policy, not just product decisions. Apple and Google. At WWDC, Apple shipped its Siri overhaul with parts powered by a Gemini partnership. EU/China rollout is delayed on regulatory grounds. Cost/commodity trend. Google cut Gemini Ultra from $250 to $200/mo and shipped 3.5 Flash; Alibaba's Qwen3.7-Plus is running at ~1/6 the per-token cost of its top tier; and open-weight models like Qwen 3.6 27B (reportedly 77.2% on SWE-bench, fits in 24GB) and Kimi K2.6 are increasingly viable for local/production use via Ollama (v0.30.8, June 12). Platform agents. Google added Managed Agents to the Gemini API, Microsoft made Copilot Cowork GA plus "Autopilot" agents, and Anthropic shipped scheduled/cron agents in beta. My take as someone building on top of these APIs: the two forces I'm watching are (1) frontier availability becoming a policy/geopolitics variable, and (2) the platforms absorbing the agent-orchestration layer that a lot of startups were building. Practically, that pushes me toward provider abstraction and keeping an open-weight fallback wired up, rather than hard-coupling to any single closed model. Curious whether others here are actually maintaining open-weight fallbacks in production, or if that's still mostly theoretical for most teams. submitted by /u/ksraj1001 [link] [comments]

How I cut my token usage in half and more (OSS, benchmarks included)

Been building Repowise for a few months now. AI coding agents are only as good as the context they get, and most of the time that context is garbage. Claude and Cursor read your files. They don't know your architecture. They don't know which files break the most and they don't know why auth got built that that way six months ago. So I built a layer that sits between the codebase and the agent. It indexes your repo into five layers and exposes them as MCP tools. I put token reduction on the title but the main premise and what I am trying to solve is so much more The five layers: Graph. tree-sitter AST into a NetworkX dependency graph across 15 languages. Leiden communities, PageRank, call resolution. Agents reason about structure instead of grepping for it. Git. Mines history into hotspots (churn x complexity), ownership, co-change pairs, bus factor. The behavioral stuff static analysis can't see. Docs. LLM wiki per module, stored in LanceDB, rebuilt on every commit so it stays in sync. Hybrid search (FTS + vector). Decisions. Architectural decisions mined from 8 sources, linked to graph nodes, with supersedes/refines/conflicts edges. Intent context, not just code. Code Health. The new one, and the part I'm most proud of. 25 deterministic biomarkers per file, 1-10 score. McCabe, brain methods, LCOM4, god classes, clone detection, untested hotspots. Zero LLM calls, runs in under 30s on a 3k-file repo. The health score isn't hand-tuned. Weights are calibrated against a real defect corpus. And it predicts bugs: 0.74 mean ROC AUC across 21 repos and 9 languages at finding files that go on to get bug-fixes. Survives controlling for file size, so it's not just flagging the big files. Ran it head to head against CodeScene on the same 2,770 files. Repowise ranked 2.3x the defects under a fixed review budget (Popt 0.607 vs 0.462, recall 0.173 vs 0.074). All paired tests, methodology and CIs in the repo. Two more deterministic signals on the same index: Change risk. Score any commit or PR range 0-10 for defect risk from the shape of the diff. PR mode flags will_break, missing_cochanges, missing_tests. Agent provenance. Attribute commits to the AI agents that wrote them. See how much of your codebase an agent produced and whether that code is a low-health hotspot owned by one person. On agent efficiency: paired SWE-QA runs with vs without the MCP tools. Loading a commit's context costs 2,391 tokens through Repowise vs 64,039 raw. 27x fewer. Across benchmarks, agents read 69-89% fewer files and make 49-70% fewer tool calls at parity answer quality. There's also distill, which compresses noisy command output before the agent reads it. pytest with 11 failures goes 3,374 -> 1,317 tokens, all 11 failure lines kept. git diff over 30 commits goes 62,833 -> 8,635. Every omission is reversible with an inline marker. 9 MCP tools total, works with any MCP-compatible agent. Local web UI to explore the graph, docs, health, and risk yourself, self-hostable, 100% local with BYO key. ~2.5k stars on github Repo: https://github.com/repowise-dev/repowise Dogfooding: https://repowise.dev submitted by /u/Obvious_Gap_5768 [link] [comments]

Artificial Analysis added a new tag for not currently available models for Fable

I just noticed codex and GPT 5.5 are near Fable level in this benchmark tho from my experience GPT 5.5 is so good to follow instructions but not as creative as Fable. Fable was just so convenient to work with like it reads my mind and even takes extra steps submitted by /u/HimaSphere [link] [comments]

Fable 5 added to the Artificial Analysis Coding Agent Index... barely 1 point ahead of GPT-5.5 ???

https://preview.redd.it/z0vkpnmp9s6h1.png?width=4640&format=png&auto=webp&s=7bb14d4d04d6cd15caf5aacc1d3c49512b7e7fd8 Artificial Analysis just added Claude Fable 5 to its Coding Agent Index (a composite average of pass@1 on DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA). The result: Claude Code + Fable 5 (max) takes the lead with a score of 77, sitting just ahead of Codex + GPT-5.5 (xhigh) at 76. A single-point difference. Seriously? Fable 5 is being marketed to us as a brand-new tier above Opus—the famous "Mythos" level yet it only edges out GPT-5.5 by one point on a composite index. To put things into perspective, Opus 4.8 (max) sits at 73 and GPT-5.5 (medium) is at 71 on the exact same chart. In other words, the gap between Fable 5 and OpenAI's flagship model is smaller than the gap between two different reasoning configurations of GPT-5.5 itself. Either these benchmarks are saturating and no longer measuring anything useful at this level, or the promised generational leap simply doesn't exist when it comes to agentic coding. Either way, it raises serious questions. And notice the frustrating detail: Fable 5's score was achieved in "max" mode, compared to "xhigh" for GPT-5.5. With both configurations pushed to their absolute limits, we are well within the margin of error. Source:https://artificialanalysis.ai/agents/coding-agents?coding-agents-performance-chart=index#coding-agents-performance-chart-tabs What do you think? Benchmark saturation or a genuine model plateau? submitted by /u/Alternative_Jump_195 [link] [comments]

Is it true

submitted by /u/Worth-Neat8209 [link] [comments]

Will you use it?

Claude is breaking the records with Mythos and Fable 5 models, what you think is just hype or it's real and based on its enormous price will you use it? And it is efficient compared to the result and cost with Codex 5.5? submitted by /u/HeadWoodpecker5237 [link] [comments]

Everyone is talking about Fable 5's benchmarks. I think they're missing the real story

The more I look at Fable 5 the more I think we're witnessing a shift that is much bigger than a single model release. For the last few years every frontier model has been competing on the same axis: intelligence. Better reasoning. Better coding. Better benchmarks. Better scores. The assumption was that whoever built the smartest model would eventually win. Fable 5 is making me question whether that assumption still holds. What caught my attention wasn't that Fable 5 is near the top of coding benchmarks. It wasn't that it sits extremely close to Mythos 5. It wasn't even the benchmark numbers themselves. It was the fact that Anthropic built an entire deployment strategy around controlling how this intelligence is used. Roughly 95% of interactions are handled directly by Fable 5 while a small percentage of requests are routed differently because the challenge is no longer whether the model can do something. The challenge is deciding when it should. That feels like a completely different phase of AI. Historically frontier labs spent most of their effort trying to make models more capable. Now it increasingly looks like they're spending enormous effort figuring out how to manage capability that already exists. The bottleneck is slowly moving away from raw intelligence and toward orchestration routing evaluation reliability and deployment. The benchmark landscape tells a similar story. Models have become so strong that researchers have had to create entirely new evaluations because older benchmarks stopped being effective at separating the frontier. Humanity's Last Exam exists largely because many leading models were already pushing past 90% on widely used evaluations. When an entire industry starts inventing harder exams because the old ones no longer tell you much that's usually a sign that the competition is changing. What's even more interesting is what happens after the benchmark. A model can score 95% on SWE-Bench and still struggle in a production environment if the surrounding system is weak. Real-world agent workflows involve retrieval memory planning tool execution API interactions validation monitoring and recovery. A single task can require dozens of decisions before it reaches completion. Suddenly the question isn't whether the model can write code. The question is whether the system can reliably execute hundreds of actions without drifting looping failing or becoming economically impractical. The strange thing is that Fable 5 may be one of the clearest signals we've seen of this transition. When a model reaches the point where the discussion shifts from "Can it do this?" to "How do we deploy this responsibly efficiently and reliably?" you've crossed an important threshold. The limiting factor is no longer intelligence alone. Five years from now I wouldn't be surprised if we look back at today's model leaderboards the same way we look back at CPU clock-speed wars. They mattered. They were important. But they ultimately became only one component of a much larger system. The companies that dominated computing weren't necessarily the ones with the fastest processors. They were the ones that built the best operating systems developer ecosystems infrastructure layers and platforms around them. Fable 5 makes me wonder whether AI is approaching the same moment. Maybe the next trillion-dollar opportunity isn't another model. Maybe it's the operating system for intelligence. submitted by /u/Bladerunner_7_ [link] [comments]

So finally it’s not AGI yet. Anyone tested it? How does it really stack against GPT 5.5 in real world coding?

submitted by /u/py-net [link] [comments]

Claude Fable 5 compared to other models and benchmarks

Claude Fable 5 is now live on BenchmarkList. We imported 34 Fable 5 result rows from Anthropic’s June 9, 2026 Claude Fable 5 and Claude Mythos 5 System Card, with the rows marked as self-reported provider results. Fable 5 looks really strong on software and agent-style benchmarks. In our BenchmarkList data it posts 95.0% on SWE-bench Verified, 80.0% on SWE-bench Pro, 84.3% on Terminal-Bench 2.1, 72.9% on CursorBench 3.1, and 85.0% on OSWorld-Verified. It also leads or sits near the top of several tool/workflow rows, including Toolathlon, AutomationBench, OfficeQA Pro Vision, and FrontierCode. The picture is more mixed outside that core. On finance, Fable 5 is strong on Anthropic’s Real-World Finance v2, with a reported 1,374 Elo and a 74% win rate vs Claude Opus 4.8, but it does not clearly lead every finance row: Finance Agent v2 has Fable 5 at 56.31%, just below Gemini 3.5 Flash in our current rows. On Vending-Bench 2, Fable 5’s best reported max-effort result is $5,680.26, competitive but below the top Opus 4.7 and GPT-5.5 rows in BenchmarkList. One big data gap: health, biology, and chemistry sections in the Anthropic system card mostly report Claude Mythos 5 rather than Fable 5. BenchmarkList therefore does not show Fable 5 scores for rows like HealthBench, BioMysteryBench, LAB-Bench 2, long-form virology, structural biology, ProteinGym, and organic chemistry. Anthropic’s launch/system-card context says Fable 5 uses conservative production safeguards, including fallback behavior for some cyber, biology, chemistry, and distillation-related requests. Compare the full set against other models on BenchmarkList. submitted by /u/davidthesong [link] [comments]

Fable 5 is live, the gateway switch makes the first run a non-event

Fable 5 just dropped and the specs are serious. Since it shares the same underlying model as Mythos 5, we are looking at SOTA benchmarks across autonomous coding, scientific research, and long-form reasoning, but with the necessary public safeguards wrapped around it. If the agentic evaluations hold up (especially the claim about running for days in a loop while checking its own work), it is going to be a non-trivial upgrade for any complex engineering workflow. Coincidentally, openrouter already has the `anthropic/claude-fable-5` model string supported, so we could trigger our first test run without rebuilding anything. Some of our other pipelines are routed through zenmux or tokenrouter, and once those list it, we'll swap those over too. The benchmark curves look great in the announcement, but the real test is seeing how it handles messy, multi-file codebase contexts over a multi-hour agent run. Rerunning our suite this afternoon. submitted by /u/Ill_Awareness6706 [link] [comments]

I feel like I’m alone. Current Anthropic models are NOT good for me, and it’s making me sad.

I can’t wait for DeepSWE to include Fable 5 in the benchmark so people can understand that Mythos is mostly hype. In the official benchmark, Opus 4.8 was supposed to be better at programming than 5.5 (SWE-bench Pro), but in one real benchmark where the model can’t cheat (I’m looking at you, Claude), it was more than 10% worse than 5.5. And once again, that was supposed to be the most powerful model in the world, completely beating 5.5 in most tasks. Like dude, shut up. Fable 5 is probably barely better than 5.5, or maybe just equal, and that’s two versions more recent than 5.5. It’s pissing me off. It’s so much more expensive and barely better, and the only reason Anthropic is even thinking of doing this kind of thing is because the AI community is full of people who don’t understand what makes a model good. From my use cases, Opus 4.8 was literally one of the worst models for me, and the most expensive. When I asked it to init a dir with Rust and Mold, it made a mistake with Mold, then told me Mold was generally broken and that it was not possible to fix, then just continued without it. When I asked 5.5 to do the exact same task, it made the exact same mistake, then fixed the path and used it. The hype around Anthropic is pissing me off so much. The models are lazy and reckless. The tools are badly implemented. Like why the fuck would you use Ink for the TUI? I don’t know, it just doesn’t feel like a lot of thought was given. People think that just because the model can create a better-looking app, it means the model is better. Like what the fuck? Yeah, congratulations on your good interface, but for me, using it in sensitive environments, I can’t fucking trust any Claude model. I keep seeing people with OpenClaw and Claude Code and whatever, launching agents to do all their work, and I’m like, great, really great, and I can’t fucking trust it to init my project. And for anyone saying it’s user error, I bought the Max plan, used the most powerful model for basically a year, and my results were consistent. I wasn’t just saying “init my dir.” I was doing prompt engineering, custom tools, when Anthropic was allowing it, kinda, with Pi coding agent, then custom instructions. I’ve tried everything, and the only thing all Claude models are good for is telling me to go to sleep. And guess what, I have alarms for that. submitted by /u/askmdev [link] [comments]