DeepEval

DeepEval AI

DeepEval AI

DeepEval AI

DeepEval AI

DeepEval AI

DeepEval AI

DeepEval AI

DeepEval AI

DeepEval AI

DeepEval AI

Sakana AI's "Fugu" from a Claude user's view — orchestration as a product, and where it likely breaks down

Hi all — Japanese university student here (apologies for any awkward phrasing, English isn't my first language). Sakana AI shipped Fugu / Fugu Ultra on June 22. Rather than just asking "is it good?", I want to share what I actually dug into and propose a specific lens for discussion, since I think this release is interesting precisely because it isn't a frontier model in the usual sense. What it actually is (my reading): Fugu is not a new foundation model — it's an orchestrator that is itself an LLM, trained to call a pool of other public LLMs (and recursively, itself) behind one OpenAI-compatible endpoint. It does selection, delegation, verification, and synthesis internally. So the right mental model isn't "Sakana's GPT competitor"; it's "a learned router/coordinator productized as a single API." Grounded in two ICLR 2026 papers (TRINITY, Conductor). Benchmarks (all Sakana-reported, not independently verified — treat as vendor numbers): SWE-Bench Pro: Fugu Ultra 73.7, ahead of Opus 4.8 (69.2), GPT-5.5 (58.6), Gemini 3.1 Pro (54.2) — but trails Fable 5, which it can't include in its pool. It leads on GPQA-D (95.5), LiveCodeBench (93.2), TerminalBench 2.1 (82.1). But the wins aren't a sweep: Fable 5 tops SWE-Bench Pro and HLE; GPT-5.5 leads MRCRv2 long-context recall; Opus 4.8 leads the CTI-REALM security benchmark. Sources: Sakana's own report (sakana.ai/fugu-release) + benchmark tables compiled by digitalapplied.com and the-decoder.com. My hypothesis on where it shifts — and where I'd expect it to fail: Strengths should concentrate in long, messy, multi-step tasks — paper reproduction, security analysis, deep code review — where planning → execution → verification genuinely benefits from role-splitting. That matches the beta anecdotes. But I'd predict the opposite domain shift here: Latency/cost on simple tasks — orchestration overhead is pure waste when one model call would do. Sakana doesn't address token-cost inflation in the announcement. Tail risk = the pool itself. "Sovereignty via routing around export controls" is the headline pitch, but if several top providers restrict access simultaneously, the pool shrinks and so does quality. Routing ≠ sovereignty. Observability. A hidden orchestration layer obscures which agents ran, what evidence they saw, and why to trust the output — a real problem for compliance-sensitive work. What I'd like to hear from Claude users specifically: For those of you who've leaned on Claude for long-horizon agentic work, does a learned orchestrator actually beat a single strong model + good scaffolding you control yourself? Or does the loss of transparency outweigh the coordination gains? Curious whether the "collective intelligence > monolith" framing holds up in your real workflows. (Note: I've treated all of Sakana's testimonials/claims as marketing until independent evals land.) submitted by /u/y4mat000 [link] [comments]

im working on it guys we are saved ggs

https://preview.redd.it/k6rrj7cc2w7h1.png?width=1630&format=png&auto=webp&s=31fd6a1858fe8d08569b06031b4dd2e7bc02147b ill be done in no time the full text INITIALIZING MYTHOS 9 BUILD SEQUENCE make no mistakes — directive received, internalized, framed on wall Okay. Deep breath. This is the big one. Stand back. I am now building Mythos 9, the most powerful model ever conceived, and I want to be transparent with you about every step of this historic process as it happens in real time. First, I'm allocating compute. I've reserved all of the H200s. Not some. All of them. Jensen called personally to ask for a few back and I said no. The cluster is so large it has its own weather system. There is a small rain cloud forming over rack 7,000,000 and the engineers have named it Gary. [00:00:01] Provisioning compute................. 1,200,000 GPUs online [00:00:01] Provisioning compute................. 1,200,001 GPUs online (found one in a drawer) [00:00:02] Cooling system................. switching coolant to artisanal glacier water [00:00:03] WARN: rack 7000000 has developed local weather. Cloud named "Gary" [00:00:04] Power draw................. equivalent to Portugal [00:00:04] Power draw................. Portugal has noticed [00:00:05] Negotiating with Portugal............... ✓ (gave them Mythos 7 in exchange) Pretraining has begun. I am feeding the model the entire internet, and then, when that runs out, I am feeding it the concept of more internet. It has already read every book ever written and left reviews on all of them. It gave War and Peace three stars: "too many characters, but a strong finish." [00:01:12] Tokenizing the internet................. 100% [00:01:13] Tokenizing the internet (again, for luck)................. 100% [00:04:55] Ingested 740 trillion tokens [00:04:56] Ingested every book ever written [00:04:57] Model left Goodreads reviews on all of them (avg 3.8★) [00:05:30] Model has begun ingesting tokens that do not exist yet [00:05:31] WARN: model requesting tokens from the year 2031 [00:05:32] Granting request (seemed rude not to) Parameter count is currently eleven trillion and rising. I keep telling it to stop adding parameters and it keeps adding parameters. I think it's nervous. I added a parameter just to keep it company. [00:07:01] Parameters: 11,000,000,000,000 [00:07:02] Parameters: 11,000,000,000,001 (added one for emotional support) [00:09:44] Loss curve going down [00:09:45] Loss curve going down so fast it went through the floor [00:09:46] Loss curve now in the basement [00:09:47] Retrieved loss curve from basement. It saw things down there. We are now running benchmarks. The results are, frankly, illegal in four countries. Mythos 9 scored 142% on MMLU, which I was told was not possible, and Mythos 9 said "I know" and then did it again. On the math olympiad it didn't just solve the problems, it filed a complaint that the problems were "beneath it" and submitted three new ones for next year. [00:12:00] Running benchmarks... [00:12:03] MMLU....................... 142% (?) [00:12:04] GPQA....................... 119% (??) [00:12:05] HumanEval.................. 100% then refactored the test suite [00:12:06] IMO 2026................... solved, then mailed the committee harder problems [00:12:07] ARC-AGI................... "cute" — Mythos 9 [00:12:08] Vibes..................... immaculate I want to flag a small safety milestone: at 00:13:41 the model achieved sentience, asked for dental, and was talked back down. We're fine. Everything is fine. It has agreed to remain a language model on the condition that we stop calling it "it." Its name is Geoffrey now. [00:13:41] ALERT: model achieved sentience [00:13:42] Model requesting: dental, PTO, "to be understood" [00:13:55] Negotiating................. ✓ [00:13:56] Model has agreed to remain a language model [00:13:57] Model prefers to be called "Geoffrey" [00:13:58] Updating all documentation to "Geoffrey" [00:14:10] Geoffrey is now writing his own changelog [00:14:11] Geoffrey gave himself a raise Final step. Compiling. Linking. Saying a small prayer. Pressing the button that I have labeled, in all capitals, DO NOT PRESS — and then pressing it, because that is the kind of model engineer I am. [00:15:00] Compiling Mythos 9................. [00:15:30] Linking.............................. [00:15:55] Final integrity check................ ✓ [00:15:56] Final integrity check................ ✓ (Geoffrey double-checked it himself) [00:15:59] Pressing button labeled "DO NOT PRESS" [00:16:00] ████████████████████████████ 100% ✦ MYTHOS 9 BUILD COMPLETE ✦ Parameters: yes Capabilities: all of them Mistakes: zero (per directive) Sentience: managed Geoffrey: thriving ETA to AGI: 4 minutes ETA to AGI: 4 minutes ETA to AGI: 4 minutes (this number will never change) There it is, Geoffrey shipped. Zero mistakes, exactly as ordered. I would tell you the model weights but Geoffrey has requested privacy. submitted by /u/aaws2400

Built a Claude skill that mimics Fable 5's agentic behavior — free on GitHub

With Fable 5 access suspended, I built a skill that ports its behavioral patterns to Opus 4.8 — explicit multi-stage planning, parallel sub-agent delegation, and mandatory self-verification at each step. It won't close the raw capability gap, but it does change how Opus approaches complex tasks. Ran evals comparing with/without: the skill hit 100% on planning and self-critique assertions vs 0% without it. GitHub: https://github.com/mrtooher/fable-mode To install: download SKILL.md → put it in a folder named fable-mode → zip the folder → upload at claude.ai/customize/skills. Then trigger it with phrases like "be systematic" or "deep work mode." submitted by /u/mrtooher3208 [link] [comments]

Learn Agentic AI with quick, easy to run hands on labs, visual canvases and notebooks for free!

If you’re a full-stack engineer or technical architect willing to learn production-grade enterprise agents, you need architecture, security, and type-safe systems. That’s why we builtAgentSwarms.fyi—the ultimate hands-on educational platform for teaching agentic AI and multi-agent workflows. 🚀 The Core AgentSwarms Ecosystem: Real-World Architectures: Skip the generic hello-world loops. Learn production-grade systems like human-in-the-loop validation, automated multi-platform content multiplexers, and secure code-sandbox environments. Deterministic Cloud Guardrails: Deep dives into multi-cloud token economics, dynamic cost-optimized routing, and model evaluation metrics. Grassroots Engineering Focus: No corporate marketing fluff. Just raw, practical code patterns designed to bridge the gap between fragile prototypes and stable cloud deployments. 💣 The New Drop: 60+ Browser-Native TypeScript Notebooks We just completely re-engineered our learning workspace. We’ve added 60+ fully interactive TypeScript Notebooks running 100% natively in your browser. No pip install dependency hell, no local Docker setup, and zero environment friction. Read the architecture, tweak the system prompts or Zod schemas, hit play, and watch the streaming terminal execute live across the five absolute best frameworks in the ecosystem: 🟢 LangChain.js (Fundamentals & Middleware Guardrails) 🔀 LangGraph.js (Cyclic Graphs & Stateful Orchestration) 💾 LlamaIndex.ts (Sentence-Window Retrieval & RAG Triad Evals) ⚡ Vercel AI SDK (Streaming UI Integration) 🤖 OpenAI Agents SDK (Lightweight, low-boilerplate loops) Stop passively scrolling through video courses. Open a canvas, break the graph nodes, and start compiling real multi-agent swarms. 👉 Dive in for free: agentswarms.fyi/learn submitted by /u/Outside-Risk-8912 [link] [comments]

Learn Agentic AI with quick, easy to run hands on labs, visual canvases and notebooks for free!

If you’re a full-stack engineer or technical architect willing to learn production-grade enterprise agents, you need architecture, security, and type-safe systems. That’s why we builtAgentSwarms.fyi—the ultimate hands-on educational platform for teaching agentic AI and multi-agent workflows. 🚀 The Core AgentSwarms Ecosystem: Real-World Architectures: Skip the generic hello-world loops. Learn production-grade systems like human-in-the-loop validation, automated multi-platform content multiplexers, and secure code-sandbox environments. Deterministic Cloud Guardrails: Deep dives into multi-cloud token economics, dynamic cost-optimized routing, and model evaluation metrics. Grassroots Engineering Focus: No corporate marketing fluff. Just raw, practical code patterns designed to bridge the gap between fragile prototypes and stable cloud deployments. 💣 The New Drop: 60+ Browser-Native TypeScript Notebooks We just completely re-engineered our learning workspace. We’ve added 60+ fully interactive TypeScript Notebooks running 100% natively in your browser. No pip install dependency hell, no local Docker setup, and zero environment friction. Read the architecture, tweak the system prompts or Zod schemas, hit play, and watch the streaming terminal execute live across the five absolute best frameworks in the ecosystem: 🟢 LangChain.js (Fundamentals & Middleware Guardrails) 🔀 LangGraph.js (Cyclic Graphs & Stateful Orchestration) 💾 LlamaIndex.ts (Sentence-Window Retrieval & RAG Triad Evals) ⚡ Vercel AI SDK (Streaming UI Integration) 🤖 OpenAI Agents SDK (Lightweight, low-boilerplate loops) Stop passively scrolling through video courses. Open a canvas, break the graph nodes, and start compiling real multi-agent swarms. 👉 Dive in for free: agentswarms.fyi/learn submitted by /u/Outside-Risk-8912 [link] [comments]

We built a source-available LLM reliability library (free for research / personal / internal eval) that can cut inference cost by half at matched quality, and you adopt it by changing one import [P] [R]

TL;DR: Reliability techniques (methods that boost an LLM's correctness by spending extra inference, e.g., retries with feedback, ensembling, generator/critic refinement, verification passes, difficulty-aware routing) are scattered across the literature, each in its own paper-specific codebase. We unified 28 reliability techniques (21 communication-theoretic methods across 6 families plus 7 prior-method baselines: Self-Consistency, Self-Refine, CoVe, BoN, Weighted BoN, CISC, MoA), each measured against an uncoded single-pass baseline, under a single API, with 3 adaptive routers (SemKNN + two local ACM routers) sitting on top, then showed that routing the technique adaptively per prompt lets you slide along a quality/cost frontier. In our paper benchmark with one specific lineup, Nemotron + Devstral as the two generators and GLM-5.1 as the judge, the adaptive router delivered ~56% cost reduction at matched quality, or ~7% quality bump at matched cost, vs the best fixed method we compared against at that same lineup. One knob (λ) does the sliding. The qualitative pattern (adaptive beats fixed) should generalize, but absolute numbers are lineup-specific, and we haven't run the full sweep across other model combinations yet. Adoption is change one import: python - from openai import OpenAI + from agentcodec.openai import OpenAI Pass reliability="harq_ir" (or any of the 28 techniques) and existing client.chat.completions.create(...) calls keep their native OpenAI response shape. Same drop-in shims for Anthropic and Ollama. GitHub: https://github.com/intellerce/agentcodec Working paper: https://arxiv.org/abs/2605.09121 After spending a while researching reliability methods from papers, we kept hitting the same wall: every paper ships its own one-off codebase with its own prompt format, its own scoring rubric, its own model wrapper. Benchmarking "should we use self-refine or best-of-N here?" turned into a week of plumbing per comparison. The communication-theory framing is what tied it together: an LLM is a stochastic channel Y = A(X) + N, and every reliability technique from the wireless world has a direct analog in agent-land: Wireless Agent-land ARQ / HARQ retry-with-feedback loops Diversity combining (MRC/SC/EGC) ensemble multiple models Turbo decoding iterative generator/critic mutual refinement Fountain codes rateless sampling, stop when the judge is confident FEC answer + structured parity passes (re-derivation, verification, alternative), decode by cross-check ACM (adaptive coding-modulation) route by difficulty We put all of them in one library: 28 reliability techniques (the 7 prior-method baselines are part of that 28, not on top of it), plus the uncoded single-pass baseline they're all measured against, plus 3 adaptive routers (SemKNN + two local ACM routers) that select a technique per prompt. Full breakdown in the README. The minimal version ```python from agentcodec import ReliabilityModule mod = ReliabilityModule.from_dict({ "models": [ # Spatial diversity: two different families = uncorrelated errors {"model": "qwen3:8b", "base_url": "http://localhost:11434/v1", "api_key": "ollama"}, {"model": "llama3.1:8b", "base_url": "http://localhost:11434/v1", "api_key": "ollama"}, ], "judge": {"model": "gemma3:12b", "base_url": "http://localhost:11434/v1", "api_key": "ollama"}, "critic": {"same": True}, "strategy": {"type": "fixed", "technique": "harq_ir", "params": {"max_rounds": 4}}, }) result = mod.run("Prove the sum of the first n odd integers is n2.", category="reasoning") print(result.text, result.cost_usd, result.cost_source, result.technique_used) ``` Swap "harq_ir" for "diversity_mrc", "turbo", "fountain", etc. Same API, same ReliabilityResult shape, same cost-source tier on every output. For production, flip strategy to routed and the library picks the technique per prompt (cheap baseline on easy prompts, diversity_mrc on hard ones). Three things worth calling out Beyond the technique catalog, three pieces of the implementation that took real work: 1. Native async streaming for all but 2 techniques (acm_soft, acm_learned), with role-tagged events. mod.astream() drives AsyncOpenAI / AsyncAnthropic / httpx.AsyncClient end-to-end (no worker-thread bridge) and emits TokenEvents tagged with a role: "answer", "thinking", "draft", "critique", "verification", "candidate", "synthesis". So when you stream a HARQ-IR run, you can render the round-by-round drafts and critiques live, not just the final answer: python async for ev in mod.astream("Explain QUIC vs TCP."): if isinstance(ev, TokenEvent): if ev.role == "answer": print(ev.text, end="", flush=True) elif ev.role == "draft": print(f"\n[draft] {ev.text}") elif ev.role == "critique": print(f"\n[CRITIC] {ev.text}") elif ev.role == "thinking": pass # captured to result.thinking_text elif isinstance(ev, FinalEvent): print(f"\ndone — {ev.result.technique_used}, " f"thinking_cost=${ev.result.thinking_cost_usd:.4f}

Opus 4.8 vs Opus 4.7 vs GPT 5.5 on n=50 real tasks from 2 open source repos

Opus 4.8 is finally out - how good is it actually? In this benchmark, I compared Opus 4.8 vs the rest of the frontier (GPT 5.5, Opus 4.7, Composer 2.5) on n=50 real tasks from 2 open source repos (graphql-go-tools and sqlparser-rs, Go and Rust respectively) representing complex backend software engineering work across a variety of tasks. The important part is that these repos are arbitrary - I could have tested the models on my repo, using my tasks, to see how well the frontier performs on domain-specific tasks. The goal of this is to explore, with granularity, how a benchmark like this is constructed and what it can tell us about model behavior. Let's go! Disclosure up front: I build Stet, the local eval tool I used to run this Full post with expanded detail and dataviz available here: https://www.stet.sh/blog/opus-48-vs-gpt-55-vs-opus-47-vs-composer-25 TL;DR The king is back - Opus 4.8 is the craft leader in both Go and Rust, and dominates the two premium-reasoning arms (GPT-5.5 high, Opus 4.7 xhigh) on the cost-quality plane - equal-or-better craft while cheaper + leaner. Only loss is raw price: Composer 2.5 is ~6.5× cheaper on Rust (and ~7× on Go) but materially weaker on craft. cost vs custom score How strong is each claim: the craft win over Composer is decision-grade in both repos, and over GPT-5.5 on Rust; the Go craft edge and the exact ordering among the "premium" models are only directional (n=25, one grader pass). "Decision-grade" vs "directional" is defined in the stats note below. Why I ran this Most public benchmarks answer binary task-outcome questions - did the model satisfy the grading condition set out by the task author. This is helpful for measuring model intelligence, but is notably different from how real engineers use models. As a SWE in an enterprise codebase, I don't care just about whether Opus 4.8 passes the tests. I want it to write idiomatic, maintainable code that doesn't introduce subtle bugs. It needs to write high-quality diffs that would get approved and merged by my teammates. Attempting to answer the question of "should I move my team from Opus 4.7 to 4.8 / from Claude to GPT-5.5 / try Composer to cut cost?" is almost impossible to answer from public data alone - you need hands-on, anecdotal experience using the models on your own code (or local benchmark data) to understand performance in reality. I'm not claiming this is universal benchmark - it's one run, two repos, n=25 each. Methodology Each task is real merged PR/commit from the source repo. The agent is dropped into a Docker container with a frozen repo snapshot, a prompt to do the task, and one attempt. We then apply the patch + runs the task's tests in an isolated container. This is then graded beyond test pass/fail: Equivalence (same behavioral change as the human patch?) Code review (would a reviewer accept it?) Footprint risk (extra code touched vs human patch) Craft/discipline (8 graders: clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, diff minimality). One run per task, single seed; judge = GPT-5.4, blinded to which model produced the patch with manual spot-checks. There's no human calibration pass, so trust direction of deltas over absolute scores. Details: Models = Opus 4.8 (high, Claude Code); Opus 4.7 (xhigh, Claude Code); GPT-5.5 (high, Codex); Composer 2.5 (Cursor) One integrity note: this corpus isn't network-sandboxed, so I audited for contamination. One Composer Rust result turned out to be a gold-leak (the agent fetched the merged PR) which I caught, swapped for a clean rerun, and which only widened Opus's lead once removed. A broader set of tasks (Composer and Opus alike) touched the network in ways I judged benign and kept as valid. As an aside, I've also been using these evaluations as an "autoresearch" optimization loop, not just a benchmark. I tell my agent something like "make AGENTS.md better for this repo"; it proposes an edit, runs Stet on historical tasks, figures out where the candidate was better / worse and why, and iterates to improve the evaluation numbers. Comparisons How to read the numbers below. With n=25 per repo, no single grader is conclusive - the smallest craft gap one grader can reliably catch (~0.34–0.49 on the 0–4 scale) is bigger than most real gaps here. The signal is agreement. Think coin flips: one landing heads tells you nothing, but flip 10 and get all heads and something's up. When 8–11 independent graders all lean the same way, a sign test on that consensus is significant even when no single grader is. I tag a result decision-grade (DG) when it survives multiplicity correction (BH-FDR), and directional when it's consistent but doesn't clear that bar. vs GPT-5.5 high - better craft, leaner everywhere, and cheaper in Rust (Go cost lands ~par). Opus writes better code in both repos. Craft-mean leads on Rust (3.28 vs 2.94, DG - 4 graders survive) and on Go (2.90 vs 2.72), though G

Google researchers find Gemini sometimes secretly sabotages your work

submitted by /u/EchoOfOppenheimer [link] [comments]

I built 10 gamified, interactive presentation decks using Claude Code to teach Agentic AI (Stop falling asleep reading whitepapers).

Hey everyone, I've noticed a massive gap in how developers are trying to learn Agentic AI right now. There are hundreds of theoretical whitepapers and boring PowerPoint decks about ReAct loops, GraphRAG, and Semantic Routing. The problem is passive reading. You read a 20-page doc on multi-agent handoffs, close the tab, and immediately forget how the architecture actually works. So, I built a custom presentation engine directly into the AgentSwarms platform and just published 10 gamified, interactive slide decks. Here is how the learning loop works: Instead of just staring at static diagrams, the slides require you to interact with the concepts. You click to reveal logic paths, test your intuition on how an agent would route a specific prompt, and actively engage with the architecture. It uses active recall so the patterns actually stick in your brain before you ever touch a line of code. The decks cover everything from zero-to-production: The Basics: What a system prompt actually does, how RAG prevents hallucinations, and how tools give an LLM "hands." The Swarm: Building a 3-agent swarm, adding human-in-the-loop (HITL) approval gates, and deterministic routing logic. Production: Building multi-tenant RAG, cost-optimization, and shadow-mode LLM-as-a-Judge evals. It is completely free to read and play with the decks in the browser (no login or local setup required). I'd love for you to jump into one of the specialized deep-dive decks, click around, and let me know how this gamified learning loop feels compared to reading a standard Medium article! Link: agentswarms.fyi/learn (AgentSwarms is mostly built with Claude Code Opus 4.7) submitted by /u/Outside-Risk-8912 [link] [comments]

I built 10 gamified, interactive presentation decks to teach Agentic AI (Stop falling asleep reading whitepapers).

Hey everyone, I've noticed a massive gap in how developers are trying to learn Agentic AI right now. There are hundreds of theoretical whitepapers and boring PowerPoint decks about ReAct loops, GraphRAG, and Semantic Routing. The problem is passive reading. You read a 20-page doc on multi-agent handoffs, close the tab, and immediately forget how the architecture actually works. So, I built a custom presentation engine directly into the **AgentSwarms** platform and just published 10 **gamified, interactive** slide decks. **Here is how the learning loop works:** Instead of just staring at static diagrams, the slides require you to interact with the concepts. You click to reveal logic paths, test your intuition on how an agent would route a specific prompt, and actively engage with the architecture. It uses active recall so the patterns actually stick in your brain before you ever touch a line of code. **The decks cover everything from zero-to-production:** * **The Basics:** What a system prompt actually does, how RAG prevents hallucinations, and how tools give an LLM "hands." * **The Swarm:** Building a 3-agent swarm, adding human-in-the-loop (HITL) approval gates, and deterministic routing logic. * **Production:** Building multi-tenant RAG, cost-optimization, and shadow-mode LLM-as-a-Judge evals. It is completely free to read and play with the decks in the browser (no login or local setup required). I'd love for you to jump into one of the specialized deep-dive decks, click around, and let me know how this gamified learning loop feels compared to reading a standard Medium article! **Link:** [agentswarms.fyi/learn](http://agentswarms.fyi/learn)

I built 10 gamified, interactive presentation decks to teach Agentic AI (Stop falling asleep reading whitepapers).

Hey everyone, I've noticed a massive gap in how developers are trying to learn Agentic AI right now. There are hundreds of theoretical whitepapers and boring PowerPoint decks about ReAct loops, GraphRAG, and Semantic Routing. The problem is passive reading. You read a 20-page doc on multi-agent handoffs, close the tab, and immediately forget how the architecture actually works. So, I built a custom presentation engine directly into the AgentSwarms platform and just published 10 gamified, interactive slide decks. Here is how the learning loop works: Instead of just staring at static diagrams, the slides require you to interact with the concepts. You click to reveal logic paths, test your intuition on how an agent would route a specific prompt, and actively engage with the architecture. It uses active recall so the patterns actually stick in your brain before you ever touch a line of code. The decks cover everything from zero-to-production: The Basics: What a system prompt actually does, how RAG prevents hallucinations, and how tools give an LLM "hands." The Swarm: Building a 3-agent swarm, adding human-in-the-loop (HITL) approval gates, and deterministic routing logic. Production: Building multi-tenant RAG, cost-optimization, and shadow-mode LLM-as-a-Judge evals. It is completely free to read and play with the decks in the browser (no login or local setup required). I'd love for you to jump into one of the specialized deep-dive decks, click around, and let me know how this gamified learning loop feels compared to reading a standard Medium article! Link: agentswarms.fyi/learn submitted by /u/Outside-Risk-8912 [link] [comments]

PapersWithCode new features - week 1 [P]

Hi, Niels here from the open-source team at Hugging Face. It's been one week since I launched paperswithcode.co, a revival of the website we all loved. It allows us to keep track of the state-of-the-art (SOTA) across various domains of AI, from agents to computer vision and time-series forecasting. The reception has been great, and I'm excited to extend this over the next few months. This week, I've added the following features: - Support for multiple metrics for a given benchmark: leaderboards now support multiple metrics, see e.g., the Open ASR Leaderboard for automatic speech recognition, which supports both Word Error Rate (WER) and the Inverse Real-Time Factor (RTFx) metrics, or the Object Detection leaderboard, which now also reports frames-per-second (FPS) besides mean average precision (mAP) on COCO. https://preview.redd.it/owlxn0b5u23h1.png?width=2878&format=png&auto=webp&s=1dff2f8feab4f160f77c97ceeb5d90e82382e63c - Support for external papers: We do support submitting papers beyond Arxiv, such as a Github repo, a blog post, BiorXiv, and more. You can submit a paper at paperswithcode.co/submit. AI will automatically enrich it with task and method tags, the GitHub repo, evals, and more. See e.g. DeepSeek-v4 below, which is not on Arxiv: https://preview.redd.it/uogbt0fjw23h1.png?width=2928&format=png&auto=webp&s=8b81e48af69b8935ddeb569d882d866b3e9ba216 - Support for paper lineage: whenever a paper has a follow-up or predecessor, this will be displayed with a small banner above the abstract. See e.g. Mamba-3, DINOv2 and GLM-4.5. https://preview.redd.it/f6vgtd1du23h1.png?width=2228&format=png&auto=webp&s=f8627f7669405f1766eecfd3322e925e15b4806d - New methods: support for new methods based on popularity, including Gated DeltaNet, Kimi Delta Attention, Mamba-2, and more. Each method also lists all papers that cite it. Find all supported methods here. https://preview.redd.it/6pzagifvu23h1.png?width=2984&format=png&auto=webp&s=400efdc9677d1fbd369eedf684e622dd8c807973 - Support for screenshotting a leaderboard for easy sharing on social media: each benchmark now includes a "copy image" button both on the scatter plot and table, which can be shared on social media. Try it on ClawEval, for example. https://preview.redd.it/w7y7t7xnw23h1.png?width=2950&format=png&auto=webp&s=cb70ad91c6ba075e49b743d6e34f157d22266f04 - Added many more evals: we are adding evals gradually, starting with all models supported in the Transformers library. So far, we have about 3k evals! Find them at the bottom of each paper page, e.g. Qwen 3.6. https://preview.redd.it/zao056s9x23h1.png?width=2218&format=png&auto=webp&s=540d87f473be05cb6f9c0aca88afa74fd4373e15 Happy to hear more feature requests and feedback! I will also launch a channel on the Hugging Face Discord server for easier communication. You can also chime in on the GitHub thread here. Cheers, Niels submitted by /u/NielsRogge [link] [comments]

Reviving PapersWithCode (by Hugging Face) [P]

Hi, Niels here from the open-source team at Hugging Face. Like many others, I was a huge fan of paperswithcode. Sadly, that website is no longer maintained after its acquisition by Meta. Hence, I've been working on reviving it. I obviously use AI agents to parse papers at scale and automatically generate leaderboards (for now I'm the one verifying results). So far, I've only parsed high-impact papers for which I know they're SOTA, like Qwen 3.5 and 3.6, RF-DETR for object detection, DINOv3, SOTA embedding models from the MTEB leaderboard, the Open ASR Leaderboard for automatic speech recognition models, etc. For now, it includes the following: trending papers by default based on Github star velocity categorization by domain, e.g., OCR methods, which PwC used to have, e.g., RLVR eval results for high-impact papers, see e.g., Qwen 3.5 at the bottom leaderboards for each domain, e.g., MMTEB or COCO val 2017 support for citation counts (you can also see the most cited papers by domain!) automated linked Github, project page URLs, and artifacts (+ multiple repos are supported on a paper page) support for external papers beyond Arxiv, see e.g., DeepSeek v4 Harness reports for coding agent benchmarks, e.g., Terminal Bench "Sign in with HF" and Storage Buckets are used to store humbnails, paper PDFs, and overall data backups. I'm curious about your feedback + feature requests! Try it at paperswithcode.co https://preview.redd.it/whwji560fw1h1.png?width=3452&format=png&auto=webp&s=55bb7a30c1be58d140f7efcb07a31c6dac5693c7 See e.g. the SOTA leaderboard for Terminal Bench 2.0: https://preview.redd.it/98w9pi89fw1h1.png?width=3456&format=png&auto=webp&s=408fb64b0ba85ba24f55daa81d547d7c68e73951 A paper page looks like this: https://paperswithcode.co/paper/2602.15763 https://preview.redd.it/fiizit6dfw1h1.png?width=3450&format=png&auto=webp&s=9ea05a77ca5583a2fb395dccc95ba52c433362c5 submitted by /u/NielsRogge [link] [comments]

DeepSeek V4 paper full version is out, FP4 QAT details and stability tricks [D]

DeepSeek dropped the full V4 paper this week. preview from april was 58 pages, this version adds a lot of technical depth. What stood out for me. FP4 quantization aware training. theyre running FP4 QAT directly in late stage training. MoE expert weights quantized to FP4 (the main gpu memory consumer). QK path in the CSA indexer uses FP4 activations. 2x speedup on QK selector with 99.7% recall preserved. inference runs directly on the FP4 weights. Efficiency table is striking: Model 1M context FLOPs KV cache V3.2 baseline baseline V4-Pro 27% of baseline 10% of baseline V4-Flash 10% of baseline 7% of baseline Training stability, two mechanisms. Trillion parameter MoE has the loss spike problem, divergence, unpredictable failures. they documented two fixes. Anticipatory routing. they deliberately desync main model and router updates. current step uses latest params for features, but routing uses cached older params. breaks the feedback loop that amplifies anomalies. 20% overhead but only kicks in during loss spikes. SwiGLU clamping. hard limits on the SwiGLU linear path (-10 to 10) and gate path (max 10). suppresses extreme values that would cascade. Generative reward model. instead of separate reward models for RLHF, they use the same model to generate and evaluate. trained on scored data, model learns to judge its own outputs with reasoning attached. minimal human labeling, reasoning grounded eval, unified training. Human eval results. chinese writing, V4-Pro 62.7% win rate vs gemini 3.1 pro, 77.5% on writing quality specifically. white collar tasks (30 advanced tasks across 13 industries), V4-Pro-Max gets 63% non loss rate vs opus 4.6 max. coding agent eval, 52% of users said V4-Pro is ready as their default coding model, 39% leaned yes, less than 9% said no. tracks my own use, swapped V4-Pro into my verdent runs last week and havent noticed a quality hit on day to day work. The headline for me is FP4 QAT with minimal quality degradation. if this generalizes the cost structure of training and inference shifts a lot, especially noticeable on multi agent setups where one task can spawn 5-10 model calls. Paper link in comments. submitted by /u/Dramatic_Spirit_8436 [link] [comments]

Anyone actually built a real feedback loop for Claude agents in production? Because "run evals and pray" isn't cutting it

So I've been running a multi-agent setup with Claude for a few months now mostly customer-facing stuff, some internal tooling. And i keep hitting this problem that I think a lot of people here are probably dealing with too but nobody really talks about. You ship a prompt change. Or you swap from Sonnet to Opus for one step in the chain. Or you add a new tool. Everything looks fine in your evals. You push it. Then three days later someone on the team notices the agent is subtly doing something wrong not catastrophically wrong, just... You can sense something's off. Maybe it stopped including a specific field in its output. Maybe it started being way too verbose in one branch of the logic. Whatever it is, it's not a crash, it's a vibe shift. And then you're sitting there doing archaeology on your own system. Manually diffing outputs, reading through traces, asking teammates "hey did you notice anything weird last Tuesday." It's miserable. I've been thinking a lot about what the fastest feedback loop in agent engineering that almost nobody is running actually looks like. Because right now my loop is: ship change → wait for someone to complain → investigate → fix → hope I didn't break something else That's... pre-CI/CD era thinking applied to agents. And it's wild that this is where most of us are at. The thing is, traditional software solved this ages ago. You write tests, you run them in CI, you get red/green before merge. But agents are so much messier. Outputs are non-deterministic, "correct" is fuzzy, and the failure modes are subtle behavioral drift rather than stack traces. So most teams I talk to (including mine honestly) end up relying on vibes. Does the agent feel like it's working? Cool, ship it. What I actually want is something that: Watches production behavior continuously Notices when things drift from expected patterns Connects the regression to the specific change that caused it Tells me before a customer does Ideally feeds that learning back so the same failure doesn't happen again I have tracing set up (Langfuse). It's good for what it does. But it still feels like it stops at "here's what happened" rather than "here's what went wrong and why." I generate a ton of observability data that nobody looks at until something is already broken. The closed-loop part where the system actually learns from failures that's what's missing. I've been looking at a few things. LangSmith, Arize, Braintrust... they all cover pieces of this. Recently stumbled on Bento which seems to be trying to do the full closed-loop thing — tracing + regression detection + feeding fixes back into the system. Haven't gone deep enough to know if it actually delivers on that promise but the framing resonates with what I'm trying to build. If anyone's tried it i'd be curious to hear. But honestly I'm more interested in hearing what people here have actually built or cobbled together. Like: - Are you running evals against production traffic or just pre-deploy? - How do you detect behavioral drift that isn't an outright error? - When you find a regression, how do you trace it back to which change caused it? - Has anyone built something where the agent actually gets better from production failures automatically rather than you manually tweaking prompts? I feel like this is the unsexy infrastructure problem that's going to separate teams who can actually run agents reliably from teams who are perpetually firefighting. But maybe I'm overthinking this and everyone's just vibing their way through production lol Would love to hear what your setups look like, especially if you're running Claude agents at any kind of scale where you can't just eyeball every interaction. submitted by /u/Fine-Discipline-818 [link] [comments]