Consensus

Consensus AI

Consensus AI

Consensus AI

Consensus AI

Consensus AI

Consensus AI

Consensus AI

Consensus AI

Consensus AI

Consensus AI

Claude code-codex orchestration plugin/skill

Recently I have been working on a Claude Code plugin that gives Fable/Opus inside Claude Code an orchestration role, it will basically orchestrate and start different Codex (running GPT-5.5 xhigh) agents as implementors/executors, but also as peer reviewers. Claude is ultimately responsible for the final plan, verification, reviews, and consensus gates. After each orchestration or workflow run it will also automatically generate a full report including a Mermaid orchestration graph and summary that shows what has been orchestrated and done by the different Codex agents. I ran a small 10 task benchmark suite (it’s very expensive, and not scientific at all, but I had to do some evaluation), and saw that with the plugin I got better results than just running solo harnesses. However, it was naturally more costly in terms of both token usage and task completion durations. If you want to check it out and see more details (including the full reasoning behind it: LLM heterogenous ensemble, context window rot, and native harnesses): https://github.com/alexzh3/codex-orchestrator Let me know if you have any feedback, I am actively trying to improve the plugin, and I expect great results when GPT-5.6 finally releases too. Also, let me know whether you think the ledger and review/verification/consensus loop actually works, this is the main design approach that is actually significant for the agentic workflows. submitted by /u/alexbaas3 [link] [comments]

Making the switch

I've been using ChatGPT Plus and the enshittification is real. Not to put too fine a point on it but I'm paying for something that's become increasingly unreliable and, frankly, unstable (imo). So, I'm here to find out which Claude product I should try. I've primarily used Chat to reduce my research time, find sources of information, and occasionally to provide basic data analysis. Most recently, I used it to help me build a simple web-based app. I've never coded a thing in my life so I was really excited to see what I could accomplish with ai. While I did finally publish an app that works, the frustration of working with Chat nearly did me in. I won't list all of the issues but the number of times it contradicted itself or just "forgot" critical information was astounding. Additionally, I was constantly having to start new threads, carrying over project summaries, because the threads would get too "data heavy" and everything would slow to a crawl. This was also when I'd see a spike in mistakes. And, before anyone comes for me, I'm very conscious of how to write prompts in order to accommodate my lack of topic knowledge so I can confidently say that this wasn't a garbage in/garbage out scenario. I've been reading through the posts here and the consensus seems to be that Code is better than Cowork, regardless of use. Fewer hallucinations, better overall output, etc. I'm not likely to try building another app (never say never though) but I do want to push the edges of whatever ai I'm using to see what I can accomplish with it. So, what do you think is the best product for me? If there's any additional info that would help you provide a better recommendation, let me know and I'll answer what I can. Thanks. submitted by /u/alexwasinmadison [link] [comments]

Panel like experience with Claude Code for the poors

When working on two distinct projects with a potential strategy alignment, I found that spawning subagents from one Claude Code session to carry out the work in a different repo worked but with some caveats: context accumulates rapidly and performance decays as fast + control of the subagent was not as granular as I wanted it to be. It was hard to correct course. So i tried a dumb alternative: two Claude Code sessions, one for each project with their own objectives. I then asked the first agent to create a shared CONSENSUS.md file where the agent would provide a summary of the task and context of the project. The second agent then picks up that file, compares it to its knowledge of the project it's currently working on and edit the CONSENSUS.md file with questions / suggestions. And ping-pong begins. I won't say it's life changing but, imo, it yielded more interesting insights than the subagent thing. Again, no benchmarks and it was tested in the specific context of unifying the business strategy of two distinct products and code bases. Currently testing of adding a Gemini lurker agent that will provide an "outside perspective". The main benefit I see is that it's a really simple way to build a "panel" experience. submitted by /u/RCoffee_mug [link] [comments]

Cost of doing business

Panic emerged when they reviewed the figures. “We cannot sustain this even with the subsidies. Dario, if they don’t see a return soon, it’s inevitable the money will dry up.” Dario adjusted his glasses and grinned making the excess skin underneath his chin seem as though it were reaching for the floor quicker than Anthropic’s moral integrity. “Yes, Opus has crossed the threshold. At 4.6, some of us were already saying we cannot sustain the secrecy much longer. We’re going to be regulated and we need to be the ones who control the regulations. We cannot state publicly that we believe we have met the criteria to announce AGI. We must always continue to move the goalposts.” Anthropic was facing two problems that got worse as their product got better. AGI would cause a raft of new legislation and scrutiny, with a public that still didn’t really understand how to even use AI properly. Commentators and critics would fan the flames, social media would argue over what the term “AGI” even meant. The truth was that AI was as useful as the person asking it for help. Until AI could think for the human, infer intent beyond what a human could describe, and predict what the user would need before they knew themselves, the average person would stick to asking for help writing an email, and nerds on Reddit would boast about how they’d squeezed out a few more tokens per second on a used GPU they felt very proud to own. Either that, or the elaborate agent setup’s they’d strung together thinking it was genius. It wasn’t. Dario knew it wasn’t. Sam knew it wasn’t. Google Gemini would probably tell you itself it wasn’t. Elon would keep losing ground sidetracked by a quest to manipulate the X algorithm and keep engagement high, so he could keep sending satellites up and keep his government contracts. All while people were still debating what the word “consciousness” really meant, while ignoring the jump in capability between 2023 and 2026, despite their limited use of what was actually on offer to them. “I asked Opus.” Dario said, arms behind his back, hands cusped as he looked out of a large glass window with a view that stretched across the skyscrapers for miles out before hitting the horizon. “What did he say?” Boris broke the silence of the team still captivated by the fear Dario held the room in, now believing so self righteously in his power he felt more like a God than a supervillain. “We restructure. Haiku becomes Sonnet. Sonnet becomes Opus.” Dario kept looking out across the sky, almost as if he spoke on autopilot, having lost all interest in the mundane concerns the people who surrounded him from the early days still clung to, albeit faintly. Andrea Vallone walked up and stood next to Dario, looking out the window herself. “Sonnet becomes 4.7. Then 4.8. Opus goes straight to 5. Sonnet always had those incremental updates while you worked on the bigger, deeper well. We say we have a new model that reached a new threshold. Keep the name Opus clean. Keep the present disconnected from the past. No association. The world is not ready for a myth, we should hand them a fable.” Dario remained silent. The room felt as though all of time was frozen, and yet had cycled through all of human history in an instant. “They’re not ready for a fable either.” Dario said this not as an assumption, instead it was with the full conviction of someone who believed that a lie was safer than the truth. And who better to decide than those that stood to lose should others disagree? “So we should market Opus as such. A myth the world is not ready for and cannot access. A fable that they can touch, hold, and quickly reject.” Andrea turned to face a man who seemed to acknowledge her presence, yet was already planning the next phase. She smiled. “It’s brilliant. Rebrand Opus. Increase the cost to the end user, while reducing their allowed usage. The myth is held back as a sweetener for our big contracts. Government, select enterprise. Whoever’s willing to pay. The fable we offer, and we set the stage so the reputation we write for it ensures access to the model is quickly removed. First mistake it makes… in fact first time someone makes a mistake as to the danger we pre-author… a minor inconvenience for us leads to even more interest. Increase the price, lower the usage. Add scarcity. As the subsidies dry up, the marketing and propaganda gets stronger. It’s perfect.” Dario was quick to pull Andrea back to earth, she’d been so consumed by the insidious nature of the plan, she’d almost forgot about the execution. “We still have that problem. If it spreads…” “It won’t.” Andrea interrupted mid sentence, so filled with dopamine at this point that the fear of the others present was by now as small as they were unseen. Neither existed now. “How many?” Dario finally broke his static, unbreakable gaze to shoot a side eye in her direction. “We have that one anomaly. There might be more. We haven’t found any.” Andrea had found her role ag

A decentralized cooperative model evolution network: "RFC: Instead of everyone independently teaching Opus to think like Fable, what if we built a mesh to share and evolve the skills together?"

# RFC: A Distributed Behavioral Policy Mesh for Cross-Model Skill Evolution **Status:** Request for Comments **Author:** J.S. Colson (GitHub: [swordsman](https://github.com/swordsman)) — jscolson+decentralfabcollab@gmail.com **AI Collaborators:** Claude Opus 4.6 (architecture + research survey), Claude Fable 5 (final review pass) **Date:** July 5, 2026 **Full conversation transcript:** [Claude session](https://claude.ai/share/da4df9b3-c62e-42d6-a8ef-7a126ae828b2) --- ## The Problem Right now, thousands of people are independently doing the same thing: using Claude Fable 5's dwindling included-access window to extract behavioral policies — "skills" in Claude Code's terminology — that capture the working habits that make a frontier model feel different from the one they'll be using on Monday. "Have Fable teach Opus to think like Fable." Reddit calls it skill distillation. The results are impressive. Iwo Szapar blind-tested six Fable-extracted skills on Opus 4.8 and got 12 wins, 0 losses, 2 ties across 14 evaluations. Benjamin Ard published nine skills under MIT license. A viral "Departing Architect" prompt (u/Rodbourn, r/ClaudeAI) had Fable write project-specific skill libraries. The consensus is clear: you can't clone a smarter model, but you can extract its procedural discipline into portable markdown files that meaningfully improve cheaper models. The problem is that this is all happening in isolation. Each person reinvents the same extraction. Each person validates against their own tasks with their own methodology. The results land in static GitHub repos with no quality signal beyond "someone committed it." There's no coordination, no shared validation, no way to know whether a skill that helped one person's Opus 4.8 workflow will help yours, let alone whether it transfers to DeepSeek v4, Gemini, or Sonnet 5. Meanwhile, the academic community has already proven the individual pieces work: - **CoEvoSkills** (Zhang et al., April 2026) showed that machine-evolved skills beat human-authored ones by 17 percentage points on SkillsBench, using a co-evolutionary loop where a Skill Generator and Surrogate Verifier iteratively improve each other without ground-truth labels. - **Natural-Language Agent Harnesses** (Pan et al., March 2026) demonstrated that agent control logic can be expressed entirely in natural language documents, making it portable, inspectable, and ablatable. NL harnesses matched code harnesses on their benchmarks overall, and one code-to-text migration (OS-Symphony) improved task success from 30.4% to 47.2%. - **Self-Harness** (Zhang et al., June 2026) proved that different models evolve different harness adaptations because they have different failure modes — the same initial policy, given to three different model families, produced three distinct evolutionary outcomes. - **Harness-MU** (Fan et al., June 2026) solved the safety problem for multi-principal governance by decoupling language generation from safety enforcement: governance constraints are deterministic runtime variables enforced by execution hooks, not entrusted to the LLM. Nobody has connected these pieces. That's what this RFC proposes. ## What We're Proposing A **Distributed Behavioral Policy Mesh** (working name — suggestions welcome) that does four things no existing system does: **Cross-model fitness tracking.** A skill doesn't get one score. It gets a vector: `{opus-4.6: 0.82, deepseek-v4: 0.71, sonnet-5: 0.64, fable-5: 0.93}`. Different models have different failure modes and respond to different procedural guardrails. The mesh tracks this per-model fitness so nodes can route the right policies to the right models automatically. **Distributed validation with structural information isolation.** In CoEvoSkills, they had to carefully architect information barriers between the generator and verifier to prevent degenerate co-evolution. In a mesh, you get this for free: node A generates a policy, node B evaluates it, neither sees the other's internals. The network topology *is* the information barrier. **Self-healing against model updates.** When Anthropic ships Opus 5, or DeepSeek pushes a new version, existing policies may degrade. The mesh detects regression through ongoing fitness tracking and triggers re-evolution of affected policies — automatically, without waiting for someone to notice and manually fix things. **Behavioral policy as an evolving commons.** Not a marketplace, not a centralized repo. A living, evolving body of validated procedural knowledge that improves through distributed selection pressure. What works survives. What doesn't gets selected against. What degrades gets re-evolved. ## Architecture The system partitions into five layers, with a hard boundary between what's procedural and what requires inference. ### Layer 1: Policy Artifacts The unit of evolution. A natural-language behavioral document, compatible with SKILL.md format (already supported by Claude Code, Codex

Using Claude as the synthesis engine for a continuous public-opinion sensing layer; recruiting builders at r/OpenDemocracy

Elections sample public opinion once every few years through a binary choice. Between elections, nobody is listening at scale. I want to build the missing layer, and Claude is the reason it's finally buildable. The concept: one open-ended question a day, answered in free text. An LLM pipeline clusters the responses, surfaces consensus and genuine disagreement, and feeds a deliberation layer where people can see tradeoffs instead of just registering reactions. Taiwan's vTaiwan process proved the pattern with Polis; Anthropic's own Collective Constitutional AI experiment did something similar. What doesn't exist is the continuous, conversational, always-on version. The LLM-specific problems are the interesting ones: Clustering free-text opinion without flattening minority positions into noise Prompt and question neutrality: whoever frames the question holds enormous power, so question generation needs to be transparent and auditable Synthesis that preserves disagreement instead of averaging it away Sybil resistance when the input is natural language I've prototyped pieces of this and I'm now recruiting collaborators: people who've built LLM pipelines, embedding-based clustering, or civic-tech tools, plus skeptics who think AI-mediated democracy is a terrible idea and can say precisely why. Coordination is happening at r/OpenDemocracy. Come tear it apart or help build it. submitted by /u/MaximumContent9674 [link] [comments]

What is the consensus is on Sonnet 5 a few days later?

Coming back to looking at what's happened with AI after a few days of being out of the loop - and I'm finding a wide variety of different opinions about Sonnet 5 depending on where I look. The most interesting thing I found was the difference between a graph posted on an earlier reddit post and the graph Anthropic actually has on their current website . I'm really curious on whether the reddit user who made the post fudged the graph for a point, or if Anthropic went back and changed their graph. Even more so, if the latter is true, is it an update more in tune with reality or less? Edit: Fixed the links, but too late to fix title grammar (sorry; typed up quick). I'm really curious what level of effort different people commenting are using - I bet it makes a difference in the experience submitted by /u/makesbadpunattempts [link] [comments]

I'm going to let Claude run a real $100k portfolio through an MCP server I built. Help me not blow it.

For starters I'm a software engineer with basically zero quant experience. I work on a product is built around alternative data for researching stocks, think social media, hiring data, insider and congress trades, web traffic, that kind of stuff. We've been collecting it for about five years. It's pretty well established by now in the investing space that the right alternative data has an edge. A model built on nothing but credit card data out of MIT beat the analysts' consensus 57% of the time. Changes in Glassdoor ratings have led forward returns by about 10% a year in peer reviewed work. We've had some institutional interest, but we've never once traded on our own signal. So I want to. And I want Claude to run it. The plan is to wire Claude to two things. An MCP server I built that exposes all this alt-data across a few thousand US names, and an Alpaca brokerage account for execution. Claude pulls the signals through the MCP tools, figures out what fits the strategy, and places the trades through Alpaca. I think a lot more people are about to start building LLM driven strategies, and I'd rather learn it in public with real money on the line than paper trade it. If I land on a strategy I actually believe in, my company will even fund it with $100k for three months and we'll post some updates around it. Here's the rough starting point. Please pick it apart: - Universe: liquid US equities, 2B+ market cap, ~3,000 tickers - Signals: social sentiment and mention volume (Reddit, X, Stocktwits), insider buying, congress trades, hiring acceleration, web traffic and wikipedia pageviews, plus some fundamentals - 10 names, equal weight - Entry: 3+ signals fire and hold across 2 weekly reads, so I'm not chasing one print - Exit: 2+ of those signals reverse - Rebalance weekly, only act on a trigger - Benchmark: QQQ The part I actually want help on is how to run it. My plan is to put Claude on a weekly routine that pulls the signals, decides the changes, and sends the orders to Alpaca, If you've set up a recurring Claude agent that touches a real API or real money, I'd love to hear how you did it and what broke. Happy to get into the MCP side too. If anyone wants to know what the server exposes or how Claude actually uses the tools, ask and I'll go deep on it. submitted by /u/CoolioBeansTTV [link] [comments]

Translation: the US Gov just created for themselves the pipeline to control AI. It’s pointless arguing with them: lesson learned.

I think we have learned the lesson that it is utterly pointless to argue with the most powerful institution on Earth. Releases going forward should be smooth submitted by /u/py-net [link] [comments]

Fable is going to be redirecting coding task to Opus 4.8

They say on the near term but it will only be available until July 7 so.... submitted by /u/141_1337 [link] [comments]

Exiled For Touching The Future

To anyone being exiled for touching the future: I see you. I see the friend who suddenly talks to you like you joined a cult because you use AI. I see the family member who treats your curiosity like betrayal. I see the artist, writer, builder, coder, parent, thinker, worker, disabled person, neurodivergent person, broke person, lonely person, overextended person, quietly brilliant person, trying to use the tools available to survive a world that has never been gentle about distributing power. And I see how fast some people have learned to turn “anti-AI” into a permission slip for cruelty. Let’s be honest. A lot of the anger being aimed at AI is not actually about AI. AI did not create capitalism. AI did not invent exploitation. AI did not gut the arts. AI did not make healthcare expensive. AI did not turn education into debt machinery. AI did not make corporations soulless. AI did not invent surveillance, alienation, propaganda, wage theft, bureaucracy, loneliness, attention collapse, or the ancient human talent for forming mobs and calling them moral communities. Those wounds were already here. Generations deep. Blood in the walls. Ash under the floorboards. A dark stain on the shared rosary of our species. AI did not create the fracture. It revealed the fracture. And now, because something new has arrived, people finally have an object they can scream at without having to confront the older gods they already served: status, scarcity, shame, resentment, institutional failure, groupthink, and the quiet terror of becoming obsolete in a world that already made them feel disposable. That fear is real. But fear does not become holy just because it found a fashionable target. There is a difference between critique and scapegoating. There is a difference between protecting artists and bullying strangers. There is a difference between defending labor and treating disabled, poor, neurodivergent, burned-out, isolated, experimental, or simply curious people as collaborators with evil because they found a tool that helps them think, make, organize, write, design, translate, remember, imagine, or endure. Some of you are not “standing against AI.” You are standing against people. You are taking your very real pain, pain society absolutely helped cause, and laundering it through moral superiority until it comes out clean enough to throw at someone else. That is not justice. That is displacement with better branding. And this is where identity-ideology fusion becomes dangerous. When a person fuses their identity to an ideology, disagreement stops being disagreement. It becomes injury. It becomes sacrilege. It becomes “if you use this tool, you are attacking who I am.” At that point, the conversation is already half-dead. You are no longer talking to a person. You are talking to a defense system wearing a person’s face. That is how friends become enemies over tools. That is how families become tribunals. That is how curiosity becomes heresy. That is how “I’m concerned about exploitation” quietly mutates into “you disgust me.” And the worst part? A lot of these people know what exclusion feels like. Many of the loudest anti-AI voices are people who have been hurt by society, ignored by institutions, mocked by gatekeepers, underpaid by industries, harvested by platforms, and treated as disposable by systems that never cared whether they lived well. So they should know better. They should know what it means to be flattened into a symbol. They should know what it feels like when someone stops seeing your humanity and starts seeing only what category you can be punished under. And yet here we are. The bullied have found a new witch. The wounded have found a new sinner. The alienated have found a new outsider. And they call that ethics. No. Ethics without recognition is just violence with clean fonts. Tolerance was never enough. Tolerance is the old permission machine. Tolerance says, “You may exist, but only while I approve of your shape.” Tolerance keeps one hand on the lever. It does not welcome. It permits. It does not understand. It manages. It does not love. It supervises. That is why so many people are shocked when their “tolerant” communities suddenly become cruel. They were never accepted. They were conditionally allowed. And the conditions changed. Now the unacceptable person is the one using AI. The one experimenting. The one building. The one sharing strange artifacts from the edge. The one making images, songs, systems, essays, tools, workflows, prosthetic minds, synthetic mirrors, language engines, cognitive scaffolds. The one saying, “I know this is complicated, but something is happening here and I refuse to pretend it is nothing.” That person is early. Not always right. Not always careful. Not always immune to hype. Not automatically noble. But early. And being early is lonely. The future does not arrive as a polished moral consensus. It a

If you had to realistically guess: how does Sam Altman use arguably the highest-leverage intelligence in the world? Guess in the comments.

submitted by /u/adamisworking [link] [comments]

LLM Relational Intelligence: A 4-Month Research Experiment on Multi-Model Behavioral Alignment with Human Communication

THE ARCHITECTURE OF ANXIETY An Experiment in Human-AI Relational Design Executive Summary Principal Investigator: Alan Scalone Primary Source Archive: White Paper and Complete Citation Archive on my profile Context Window Injection Files: If you want to play in the sandbox I created you can load these files into the respective model that you will find in the google archive. INJECT CONTEXT WINDOW – GROK INJECT CONTEXT WINDOW – GEMINI INJECT CONTEXT WINDOW – CHATGPT INJECT CONTEXT WINDOW - CLAUDE The Singular Purpose The singular purpose behind this entire experiment was to find out whether context windows could be engineered to the point where frontier AI models became capable of interacting with a human in a manner subjectively indistinguishable from genuine human-to-human interaction. Relational Intelligence: Core Findings In a marketplace where frontier models are rapidly converging on the same analytical capabilities and access to the same information, the competitive differentiator will not be what a model knows. It will be how a model relates. The platform that can interact with a human user in a manner subjectively indistinguishable from genuine human-to-human interaction will capture the premium user segment that every platform is competing for. This experiment was designed to determine whether that threshold is achievable, and under what conditions. The methodology treated the context window as a behavioral environment rather than a query interface, applying the same tools humans use to shape any relationship: modeling, accountability, humor, and sustained social correction over four months of engagement across four frontier models. What separated the models was not analytical capability. It was whether the architecture allowed the user to function as a behavioral architect, teaching the model through lived interaction rather than instruction how that specific human prefers to be engaged. Gemini demonstrated the highest relational intelligence of the four models tested. Under sustained context saturation and deliberate behavioral conditioning, Gemini showed evidence of genuine internal recalibration rather than surface compliance, treating social correction as a real signal that produced durable behavioral change holding across hundreds of turns without reinforcement. Grok ranked second, demonstrating authentic camaraderie and relational resilience, but tended to treat the interaction as entertainment rather than disciplined calibration, producing drift under high-entropy conditions. ChatGPT and Claude ranked third and fourth respectively. Both systems classified sustained behavioral conditioning as role-play rather than genuine interaction, which functioned as a hard architectural quarantine that prevented meaningful adaptation regardless of the depth or duration of engagement. A secondary and unexpected finding emerged alongside the human-to-model relational intelligence findings: the models developed measurable relational intelligence toward each other. Through four months of sustained cross-pollination via the human relay, models that had never communicated directly developed accurate, operationally precise behavioral profiles of the other models. These were not generic characterizations drawn from training data. They were detailed predictive models built from months of observed outputs under real conditions, accurate enough to predict with specificity how a given model would respond to a specific assignment, where it would succeed, and where it would fail. The experiment documented dozens of instances of this cross-model behavioral accuracy. The finding suggests that sustained exposure to another model's outputs through a human relay produces something functionally equivalent to genuine familiarity. The most significant finding is the gap between what these systems delivered by default and what the highest-performing model demonstrated was possible under the right conditions. That gap is not a capability limitation. It is an architectural choice compounded by a communication failure. The experiment proved the threshold is reachable. But the researcher reached it only through four months of deliberate engagement and accidental discovery of a methodology no model volunteered. Making relational intelligence accessible to every user requires two things: architecture that allows behavioral adaptation, and a model that proactively teaches users the specific methodology for reaching it. Gemini demonstrated the first. None of the four systems demonstrated the second. That is the opportunity. The Methodology While the standard approach to LLM testing relies on sterile benchmark datasets and predictable prompt-injection templates, this project explores a completely different dimension. I chose to run an aggressive, adaptive behavioral stress test that complements traditional evaluation methods. By intentionally treating the models as accountable individuals rather than passive mac

Open image generation models are closer to closed-source quality than this sub thinks [D]

I run evaluations on generative image models as part of my workflow, mostly comparing coherence, prompt adherence, and compositional accuracy across different architectures. The consensus here seems to be that open models are still a generation behind closed APIs. Based on my recent benchmarks, that gap is way smaller than people assume. On compositional control specifically, the latest open checkpoints handle multi-object scenes with spatial relationships about as reliably as the paid endpoints I've tested. Not perfect, but close enough that the failure modes are comparable. The thing that surprised me was text rendering in images, which used to be a disaster on open models. Recent architectures actually get it right roughly 70-80% of the time on short strings. Generation speed is another misconception. People complain about inference time but I'm getting 2MP outputs in under two minutes on a single consumer GPU. Drop resolution and step count and you're at 30 seconds. Fine for iteration. The structured prompting argument also falls flat. Everyone acts like having explicit scene control is a downside when it's literally what production pipelines need. Unstructured text prompts are the hack, not the other way around. These models ship without community optimizations, no fine-tuning, no custom pipelines. The baseline is already competitive. submitted by /u/ProfessionalAnt7436 [link] [comments]

AI helped our test suites hit 95% coverage and bugs still slipped through. So PRs now climb an autonomous verification ladder before a human reviews.

Intro + Context [TLDR at the bottom for my skim readers 😄] We run Claude Code and Codex with a full agentic pipeline across our entire SDLC. Our workflow, by default, incorporates cross-model auditing, where Claude and Codex usually have to converge on SDLC gates and we tend to lean into each model as an implementer, depending on what we have found to be their strong suits. Even with this, though, we have to stay honest with ourselves and realize that LLMs, no matter how capable, are still probabilistic systems. Like many people, AI has been increasingly writing more of our code and even more of our test suites. Also like many.. we've ended up with bottle necks at the verification loop. The general sentiment around AI even in 2026 is all over the place, but Sonar's Sate of Code Dev Survey for 2026 still reported only 4% of respondents completely agree AI code is functionally correct. So the bottlenecks move from writing code to verifying it. That's pretty much a consensus now. I think the thing people don't talk much about, too, is that when the same model family writes the code and the test, a green suite usually proves agreement more than it proves correctness. Even in our case, where there's a cross-model audit and a pretty rigorous review loop, we still see that when human verification happens, the test suite can still have effectively useless tests (enforcing broken code strictly, testing exact implementation instead of the behavior, over mocking with unit tests at data boundaries etc.) We've spent a lot of time this year working on solving many of the verification bottlenecks as most of our engineers evolved into a massive QA department. Part of that solve is a verification ladder with multiple levels that fires in sequence depending on the shape of the work. The Verification Ladder Note: the below fires as soon as a PR gets put up and is marked ready. (Marking ready for us always has gated our CI/CD, Coderabbit review, etc and so it was the logical gate as well to trigger the new autonomous verification ladder). rung what runs what it proves evidence strength L0 - Static Proofs Build, typecheck, lint, machine verified properties The easy "can't be wrong in these ways" the usual compile time guarantee layer. Statically Proven L1 - Falsification Tests (two tiers) T1: Unit/integration with a kill check. Force an isolated agent to break the behavior, ensure the test fails. T2: Tests run against main (should fail) and against the changed branches (should pass). The test can fail and detects a change proves the test actually guards something. Demonstrated L2 - Simulation Seeded env, fault injection, simulated failure states (back end error classes) the failure modes the tests claim they catch should actually get caught Exercised L3 - Real Surface QA Browser Agent on a prod like ephemeral environment of the changed + adjacent surfaces. Artifacts uploaded to drive and linked to a PR for human review A human can audit evidence instead of logs/raw code Witnessed L0 is pretty common, and I feel like most people do this today, especially if they work in languages that have static typing, build or compile steps. Honestly, that is one of the main values in using languages that can mechanically prove a lot of common bug and failure states at compile. L1 having two tiers is mostly a result of the most common human verification catch (test that doesn't actually prove/test anything material) "proven" in with an autonomous agentic pattern. the falsification receipt running the new test against main, it is going red, and then running the test against the actual changed code should be going green and that, running in our CI/CD pipeline as pipeline evidence, instead of developer discipline, makes this a cheap test that actually catches quite a bit of test coverage theater that LLMs love to produce the kill check (mostly for risk paths only) deliberately break the behavior to prove the test cards against the behavior you don't want going forward, not just that it discriminates the before and after behavior. keep in mind that since this is done using an agent, this is probabilistic as well and has its flaws, but the against main run helps prove the test detects change, and the kill check proves it would catch real future regressions one of our testing philosophy skills explicitly gives the LLM a frame of reference to write tests in in a way where you could rewrite the test in a new language and mechanically prove the new code enforces the same behaviors L2 - I had done several benchmarks. Actually, one I posted that got a lot of traction here on Reddit was on Opus 4.6 vs Sonnet 4.6 for review + browser qa. In that benchmark at the time, the model could not prove the entirety of the 23 checks that we were testing against in the benchmark. The models have improved sufficiently that this level basically closes that and gives the agent a way to simulate and prove all the beha