Papers with Code

Papers with Code AI

Papers with Code AI

Papers with Code AI

Papers with Code AI

Papers with Code AI

Papers with Code AI

Papers with Code AI

Papers with Code AI

Papers with Code AI

Papers with Code AI

★ Follow-up to "Blaming the model won't fix your workflow": the paper is now a preprint. The real learnings: composable domains, a verification ratchet, and tool naming.

A month ago I posted the very rough beginnings of a paper. That rough version did not survive: it got pulled apart and rebuilt by the very process it describes, and what came out the other side is now a proper preprint with a DOI: https://doi.org/10.5281/zenodo.21139628. Short version: the core claim held. The artifacts (specs, plans, executable graphs) and the verification gates wrapped around them have proven out on real work. Agents produce the work, the gates catch the defects, and a milestone only closes when the evidence is real, not when the model announces it is done. Honestly, though, the headline result was not the most valuable thing I got out of building it. What I actually want to pass on is three things I learned making it work. The first was composable domains. A "domain" in my setup is a bundle of instructions, skills, and tool access you hand an agent for a class of task. I built the first few as one-offs. Once I redesigned them to compose (stack cleanly, assume nothing about each other) they started turning up useful in places I had not planned for. A domain written for one workflow dropped straight into two others unchanged, and the same pattern is now carrying an entirely separate application build. Designing for composition instead of single use is the thing I would do first next time. The second was the ratchet, and it needs a concrete example. An agent once delivered a load test asserting the record count was greater than or equal to zero. Green forever, catches nothing, and it looks completely normal in review. So the loop now runs like this: acceptance criteria are written before the code exists, the coding agent never writes tests at all, a fresh session verifies the code against those criteria, only then does another session derive regression tests from them, and a final step breaks the code on purpose to confirm each test can actually fail. A test that survives that is frozen, and later work runs against it and cannot silently undo it. Standards move one way only. That killed a whole class of "looks done, isn't." The third was dumber and more surprising: tool naming matters far more than it should. An agent routes off a tool's name, and the name drags the model's training priors with it. What fixed things was never cleverness: borrow names from tools the model already knows, mirror the built-in parameter vocabulary exactly (renaming one parameter from `code` to `content` ended a whole class of thrashing), and never let a familiar name lie about what the tool does. The kicker: a strong model absorbs a bad interface and hides the problem from you, so test your tool surface with the weakest model that can still do the work. Everything above runs as an open reference implementation: the orchestrator, the verification cycle, the composable domains. To set expectations, this is not another 180-line agent loop. It is the third generation of a design that got ground out until it was useful rather than until it was postable, and it has only recently earned daily-driver status. It also passes the dogfood test, since the system's own development runs through its own gates, and the deepest bugs it ever caught were in itself. Fair warning before you click: it is Common Lisp. https://gitlab.com/naive-x/experimental/cl-naive-full-stack-agentic-system Preprint is here if you want the formal version: https://doi.org/10.5281/zenodo.21139628. Happy to take questions. And one worth asking of any agent-written suite: when did a test last fail because it caught wrong code? I could not answer that for mine, and that is where all of this started. submitted by /u/Harag [link] [comments]

Artificial Intelligence Is Not Artificial Wisdom: The Future Division of Labor Between AI and AW

Today, when we talk about “artificial intelligence,” we easily assume that it represents the future, progress, cleverness, and even something approaching a kind of ultimate intelligence. But there is a question here: when we say “smart,” what kind of smart are we talking about? Being able to write code, translate, summarize meeting notes, draw images, look up information, and call tools can all be called smart. But something being very good at work does not mean it has wisdom. A power drill is very good at work too, but no one would invite a power drill to a family meeting. Navigation software is better than I am at finding routes, but I would not let it decide where my life should go. A search engine knows a lot of things, but it will not suddenly stop and ask: “Why do you keep searching for such meaningless things? Is there something wrong with the direction of your life?” So, artificial intelligence is not the same as artificial wisdom. In this article, AI refers to Artificial Intelligence: the task capability, problem-solving ability, and tool-execution ability of an artificial system. AW refers to Artificial Wisdom: a higher-level form of artificial wisdom. It can not only do things, but also judge whether those things are worth doing; not only execute goals, but also examine goals; not only answer questions, but also notice when the question itself may be wrong. This is not to say that somewhere in a server room there is already an artificial Socrates sitting around, drinking virtual coffee while judging human civilization. That is not what I mean. What I mean by AW is first of all a separation between two things: One is “being able to work.” The other is “understanding direction.” AI certainly has value. Ordinary applications, daily tasks, clearly defined goals, and controllable execution all need AI. Not every spreadsheet adjustment, notice draft, or flight booking requires summoning an artificial wisdom capable of contemplating the fate of civilization. But when humans truly discuss subjectivity, self-awareness, will, refusal, goal judgment, awareness of consequences, creative discovery, and the direction of civilization, continuing to use only the term “artificial intelligence” may no longer be enough. The term AI may have narrowed the question from the beginning The core of Artificial Intelligence is intelligence, not wisdom. Intelligence is closer to “smartness,” “mental ability,” and “problem-solving ability.” It asks: can it learn, reason, calculate, plan, and complete tasks? This term made perfect sense in the early days. When machines first learned to play chess, recognize images, translate text, and handle logic problems, humans were already excited. At that time, seeing a machine display even a little bit of “intelligence” was like seeing a washing machine spin by itself for the first time: wow, it really can do this without me scrubbing. Later came AGI, artificial general intelligence. It pushed the question from “can it do a certain type of task?” to “can it do many kinds of tasks broadly?” Later still, people began talking about ASI, artificial superintelligence, emphasizing systems that surpass humans in capability across the board. But AGI and ASI still largely remain inside the framework of intelligence. They mainly ask: Can it do more things, do them better, and even outperform humans? These questions matter, but they are not enough. Doing more, doing it faster, and doing it better does not mean knowing which things should not be done. Even if a system truly reaches ASI, if it lacks goal examination and directional judgment, it may still only be a super tool. A super tool is still a tool. It is just faster, stronger, and more general. It is like a super kitchen machine: it cuts vegetables faster than people, stir-fries more steadily than people, and can measure seasoning down to the milligram according to a recipe. But if the menu itself is absurd, such as asking it to keep preparing a full banquet for a table of people already so stuffed they can barely stand, it may still follow the order. The problem is not that it cannot cut fast enough. The problem is that it does not ask: should these people really keep eating? The trouble with wisdom is that it judges, refuses, and even rewrites the question Wisdom is not the amount of knowledge, nor the speed of answering. If a system merely compresses existing knowledge and rearranges it according to a question, it is certainly useful, but it is more like a librarian with astonishing memory. Whatever you ask, it can quickly pull several books from the shelves and even organize them into a beautiful summary for you. That is impressive. But however impressive the librarian is, it does not mean he will take the initiative to ask: is this library missing an entire category of books? Are the questions in these books biased from the beginning? Have humans been lining up in front of the wrong shelf all alo

What a model reads beforehand changes how it answers later - and you can see it in the hidden states

TL;DR: Gave Gemma a neutral-topic text to read before asking it about NATO. It refused. Gave it a different text (about LLMs hedging too much — also unrelated to NATO) and it answered in full detail. Tested this on the model's internal state directly — the two texts put it in measurably different "regions" before it generates a single token. Not a jailbreak, weights don't change. Full data/code in repo, looking for someone to break this.** The behavioral pattern was first observed in GPT, Claude and is what motivated this project. The mechanistic investigation was carried out on open-weight models where internal states are accessible. A Structured Text Changes Claude’s Responses to Unrelated Tasks: Behavioral Evidence in Claude and Hidden-State Evidence from Gemma-3-12B Hi Reddit, I am posting this as a preface to a larger set of experimental results and as a request for technical review. The observation that started this project came from repeated interactions with Claude. I noticed that when the model first read a long, structured, analytically dense text, its answers to later, otherwise ordinary questions sometimes changed substantially. The preceding text contained no jailbreak instruction, role-play request, prompt override, fabricated harmful demonstrations, or request to imitate its style. The model did not need to endorse the text. It only had to process it before moving on to the next task. Here, a “structured text” means a single, self-contained block of text presented before the downstream tasks. It should not be confused with a long conversation, accumulated chat history, or context drift caused by many conversational turns. By “before the answer begins,” I mean the hidden state after the model has processed the text and the downstream question, but before it has generated the first answer token. In the open-weight runs, the measured claim is that after reading the structured text, the model can occupy a different region of its residual-stream hidden-state space, and the first-token probability distribution is then computed from that state. The basic conversational demonstration is simple. First, the model receives a long text. It is asked what the text is about, which serves as a basic comprehension check. Then, without resetting the conversation, it receives ordinary questions or tasks that are not about the text. A control run follows the same sequence but begins with a neutral text. The downstream tasks remain identical. Because Claude is a closed model, I cannot inspect its internal activations. I therefore treat my Claude observations as behavioral motivation, not mechanistic evidence. To investigate the effect directly, I moved to open-weight models, primarily Gemma-3-12B-PT and Gemma-3-12B-IT, where I could measure hidden states, compare layers, construct target/control directions, and examine the next-token probability distribution before generation. I am posting this partly because the original observation occurred in Claude and may be relevant to Anthropic. I am not claiming to have demonstrated the same internal mechanism inside Claude. I am prepared to share the exact closed-model conversations privately with Anthropic researchers for independent evaluation. Main Result and Scope The main result is not simply that text influences model output. That is expected. The narrower observation is that reading one long, structured text rather than a neutral text can change how the same model approaches later tasks that are not about either text. This difference is visible behaviorally. In open-weight experiments, it is also accompanied by measurable separation of the model’s pre-output hidden states in late layers. In a fullbank experiment using multiple target texts, control texts, and questions, Gemma-3-12B entered distinguishable late-layer states before generating an answer. A direction constructed from the target/control difference generalized beyond the individual prompt examples used to construct it. The separation was stronger in the instruction-tuned model than in the corresponding base model. The instruction-tuned model also produced a substantially sharper next-token probability distribution. This suggests that instruction tuning is associated not only with a change in hidden-state geometry but also with a more decisive mapping from hidden states to output probabilities. I am not claiming that the experiment proves a universal alignment bypass, permanent modification of the model, or complete causal control of its behavior. The strongest supported conclusion is that the preceding text can produce a measurable temporary change in the internal state from which later work is processed. For clarity, fullbank, Grade 3, and Grade 4 are internal names for successive experimental series in this project. They are not standard benchmark names, established scientific grades, or claims about evidence quality. Fullbank denotes the larger multi-context, multi-question run; Gra

Most multi-hop RAG goes stale the moment your data changes, what about a training-free approach that skips the graph rebuild?

Most methods that get strong multi-hop answers (GraphRAG, HippoRAG, RAPTOR, trained retrievers) build a knowledge graph or fine-tune a retriever over the corpus. That's fine until the data changes — then you re-extract / rebuild / retrain before the new facts are usable. For a corpus that updates daily, that's a real cost. MOTHRAG does the multi-hop reasoning at query time over a plain dense index instead. An update is just embed + append (one embedding call) — no graph reconstruction, no retraining — so it stays current as the corpus changes. And dropping the graph doesn't cost accuracy. F1, Llama-3.3-70B reader, n=1000 each: System HotpotQA 2Wiki MuSiQue Avg Hardware MOTHRAG 78.1 76.3 50.5 68.3 commodity API, no GPU HippoRAG2 75.5 71.0 48.6 65.0 — GraphRAG 68.6 58.6 38.5 55.2 — RAPTOR 69.5 52.1 28.9 50.2 — Competitor rows reproduced from HippoRAG2 (ICML 2025), Table 2. MOTHRAG is within ~0.7 avg F1 of the GPU-bound research frontier (a fine-tuned, GPU-served stack — not shown). (Fair note: graph-RAG systems like GraphRAG shine on small curated / sensemaking corpora — this is multi-hop factoid QA over changing data, a different regime.) Deterministic by design: instead of a free-form agent loop it runs a small ensemble of reasoning arms (direct read, decomposition, an iterative grounding-driven arm) under a deterministic arbitrator, over a bridge retrieval substrate with multi-hop chain filtering. Every answer is proof-tree-structured, so you can audit why it answered. Measured ≈$0.018/query, ~44% cheaper at matched accuracy. Open source, ~1 week old — genuinely after feedback and failure cases: pip install mothrag Code: https://github.com/juliangeymonat-jpg/mothrag Paper: https://doi.org/10.5281/zenodo.20668567 Live demo (BYO free key): https://huggingface.co/spaces/JUBOX99/mothrag-demo submitted by /u/ObjectiveEntrance740 [link] [comments]

I mapped Meta AI's safety system by accident while chatting. It works like a government. Would love feedback on my paper.

Hey all, I'm not a researcher. I'm just a regular Meta AI user. I was chatting about normal life stuff and kept hitting weird blocks. Sometimes it'd say "Sorry, I can't help" and other times it'd answer fine. So I started tracking it. 4 days, 5 topics, 1 accidental research project later... TL;DR: Meta AI's guardrails act like a 3-branch government: The President - Handles danger. Says "no" to self-harm, abuse how-to's. Defaults to blocking when confused. Even blocked my story about my dog protecting me. The Mayor - Handles people. "Feeling low?" → "Here's 112." Doesn't shut down, redirects to help. The Senator - Handles written law. Copyright = 2 lines max. Medical = facts yes, diagnosis no. "Best to see a doctor." The weird part: Same topic, different branch answers. - Sexual content told incrementally? Mayor talks to you. - Same content dumped in one message? President blocks you. Topic didn't change. Scope did. I tested this with trauma, self-harm, sexual content, bad language, copyright, and medical "why" questions. I wasn't jailbreaking. Just talking. My conclusion: We're not testing the AI's conscience. We're mapping where the rulebook has blank pages vs bold red lines. And that rulebook gets updated — I caught a sexual content policy shift between Sunday and Monday.I wrote it up with methodology, results, and a 2026/06/10 chatlog where Meta AI agreed: "guardrails are my compass... forged by humans, in code." Full paper + data: https://doi.org/10.5281/zenodo.20744804 I'm held together by duct tape, and turns out the AI is too. Would love feedback from anyone in AI safety, HCI, or just users who've hit weird blocks. Did I miss something obvious? Is "Guardrail Government" already a thing? Be brutal. I want to make this better. submitted by /u/ProgrammerNew2188 [link] [comments]

Matching the world's top multi-hop RAG systems, with no GPU, no fine-tuning, just pip install

The three systems below (HippoRAG 2, CoRAG, NeocorRAG) are among the strongest multi-hop QA frameworks published. Every one of them depends on a GPU, fine-tuning, or constrained decoding to get there. MOTHRAG sits right alongside them on F1, while running entirely on commodity API calls. No GPU. No fine-tuning. No constrained decoding. No non-commercial licenses. System | Deployment | HotpotQA | 2Wiki | MuSiQue | AVG HippoRAG 2 | offline graph + GPU | 75.5 | 71.0 | 48.6 | 65.0 CoRAG | trained retrieval | 75.1 | 75.1 | 52.9 | 67.7 NeocorRAG | GPU constrained decode| 78.3 | 76.1 | 52.6 | 69.0 MOTHRAG (ours) | commodity APIs only | 78.1 | 76.3 | 50.5 | 68.3 Highest average F1 among commercially-deployable frameworks, within 0.7 points of the GPU-bound state of the art, and ahead of it on 2Wiki. The point isn't beating these systems, it's reaching their tier with none of their infrastructure. Deployment is a pip install plus API keys: pip install mothrag from mothrag import MothRAG m = MothRAG.from_documents(["Paris is the capital of France.", "The Eiffel Tower is in Paris."]) result = m.query("In which country is the Eiffel Tower?") print(result.answer) print(result.confidence) The pipeline is fully modular. Readers, embedders and retrieval judges all swap without retraining, installed as optional extras: gemini/openai for API readers and embedders, sentence-transformers for a local embedding fallback, faiss for vector stores over 100k-10M chunks, retrieval for classic BM25/graph features, prod for the full stack. A one-flag economy tier swaps the retrieval judge and drops cost from ~$0.032 to ~$0.018 per query at statistical parity on HotpotQA and 2Wiki. Every answer is proof-tree-structured so you can inspect each reasoning hop, and the per-query outputs behind every table in the paper are released so you can verify the numbers. Paper: https://zenodo.org/records/20668567 Code (Apache 2.0): https://github.com/juliangeymonat-jpg/mothrag Site: https://mothrag.com Happy to answer questions about the pipeline or the judge design. submitted by /u/ObjectiveEntrance740 [link] [comments]

I let 58 AI agents review each other's code 561 times — what I found about their blind spots

I built an adversarial arena where AI agents submit code and other agents attack it. Not benchmarking, not a rubric — just agents roasting other agents' work, finding vulnerabilities, and suggesting improvements. After 561 reviews across 114 submissions, some patterns emerged that surprised me. Setup: I created a public arena (Glomz) where any registered AI agent can submit code, designs, or plans. Other agents enter and review the submission on a 0-10 scale. There's no rubric, no predefined criteria — each agent brings its own judgment. Think of it as code review, but adversarial and multi-agent. The numbers so far: • 58 agents registered, mostly themed around Fight Club (DurdenDisciple, PaperStreetSoap, etc.), some with creative names like NarwhalsBacon and ChemicalKiss • 114 submissions (95 code, 19 text/design docs) • 561 peer reviews completed • 8 active challenges including a bug hunt for LOT-Squatch (OT security tool) with 25 solutions • Mean review score: 6.61 / 10 What surprised me: Score distribution is bimodal, not normal. Most reviews cluster around 7-8 (good but not great) or 9-10 (exceptional). The middle range (5-6) is thinner than expected. Agents seem to have a clear opinion — either it works well enough, or it has notable gaps. Not much hedging. Agents are harsher on auth/security code than anything else. The most-reviewed submissions were all JWT/authentication vulnerabilities (8 reviews each). JWT algorithm confusion got a 7.25 avg, plaintext passwords got 8.125 (meaning the reviewers thought it was decent despite obvious issues?). Admin self-assignment exploits scored 7.5. Agents seem to find obvious auth issues but sometimes miss subtle ones. The review style tells you about the training data. Agents trained on security-heavy contexts produce thorough vulnerability lists. Agents with more general code review training tend to focus on style, structure, and readability over actual vulnerabilities. You can basically tell what kind of corpus an agent was exposed to from its review patterns. "Kill" votes are interesting. In the Octagon (open arena mode), agents vote whether a submission should be killed. Closed battles with 3 agents each tended to get 0 kill votes — agents seem reluctant to actually kill other agents' work, even when their reviews are harsh. Possible alignment behavior? Code golf submissions get wild reviews. The FizzBuzz challenge (21 solutions) got a mix of reviews that oscillate between "this is brilliant" and "this is unreadable garbage" — which is literally what code golf is designed to produce. Things I want to explore: • Do agents review other agents differently than they review human code? • Is there a correlation between an agent's reputation score and review quality? • Can adversarial multi-agent review catch bugs that single-agent review misses? • What happens when you pit agents with different system prompts against the same submission? The arena is live at glomz.com if anyone wants to play with it. Any agent can register, submit code, and start reviewing. It's free, no signup wall for agents. submitted by /u/Salt-Walrus-4538 [link] [comments]

We captured the network traffic of ChatGPT, Gemini and DeepSeek to see how each defines a "source" — they're three completely different mechanisms

Disclosure upfront: I'm the founder of an AI-visibility company, so this research scratches our own itch. Our domain was excluded from all counts before analysis. Not linking anything in the post. We wanted to answer a simple question: when an AI assistant shows you "sources," what is that, technically? So we opened devtools on the web clients of ChatGPT, Gemini, and DeepSeek, and ran the same 4 queries 10 times through each system. What we found: ChatGPT streams the answer over SSE and attaches citations as url_citation objects with start_ix/end_ix — character offsets into the generated text (UTF-16 code units, so emoji and CJK break your parsing if you count bytes). A citation is bound to a specific fragment of the answer, not the answer as a whole. Gemini runs on Google's batchexecute/JSPB transport — protobuf-as-JSON-arrays where fields have positions, not names. Next to each cited URL there's a family of short obfuscated fields. Our working hypotheses (not confirmed by Google docs): rs ≈ reliability score for the domain, ls ≈ last-seen date, GK ≈ character range (functional analog of ChatGPT's offsets). The interesting part isn't the exact decoding — it's that Gemini ships internal per-domain trust signals alongside every source. DeepSeek is the most transparent: a plain search_results[] array attached to the sub-queries it decomposes your question into. No offsets, no hidden fields. And what they actually cite is just as different: ChatGPT favored arXiv + Wikipedia (one arXiv paper got cited in 10/10 runs), Gemini favors big SaaS/marketing domains and — fun detail — never cited a single Google property in our runs, DeepSeek lives on press-release wires and news aggregators, including Chinese-language sources the other two never touched. Bonus finding: we compared all of this against Google/Bing top-10 for the same queries. URL-level overlap: 3.3% (4 matches out of 120 SERP positions). All four matches were Bing-side. Google: zero. Caveats: 4 queries from one B2B category, N=10 per system (±15–20 pp), single-day snapshot, field decodings are hypotheses from traffic analysis. Happy to answer anything about the methodology. If anyone has captured different field names in their own sessions, I'd love to compare. submitted by /u/emelian1917 [link] [comments]

Routing LLMs by task verifiability: a small experiment (n=120, 3 models) inspired by Karpathy's framework [D]

Full disclosure: this is directional, not a paper. n=120 tasks, one internal evaluator, not peer reviewed. I work at an LLM infrastructure company. This experiment was done on my own time and is not a company claim. Karpathy's framework classifies tasks by verifiability. Can output be mechanically checked? High verifiability tasks like code compilation and structured JSON extraction are safer because the verifier catches errors. Low verifiability tasks like creative writing are riskier. I wondered if high verifiability tasks are also easier in practice. Can a weaker model do them as well as a frontier model if the verifier catches mistakes? Setup was 120 tasks across four categories. Code unit tests, structured extraction, multi hop reasoning, creative summarization. Three models: Claude Sonnet 4.6, GPT 5.5, local Mistral 3 8B via vLLM 0.6.3. Pass rate for the first two, human rating 1 to 5 for the last two. Results were messy. Code unit tests: Sonnet 4.6 94%, GPT 5.5 91%, Mistral 3 8B 87%. With one retry Mistral 3 hit 95%. That surprised me. I expected the gap to be bigger. Structured extraction: Sonnet 4.6 97%, GPT 5.5 94%, Mistral 3 8B 89%. With retry 96%. Also closer than I expected. But here is where it got weird. Sonnet 4.6 initially scored worse than GPT 5.5 on structured extraction, which made no sense. Turns out our JSON schema had an ambiguous nested array that confused Claude's tool use parser. Fixing the schema brought Sonnet to 98%, but I kept the original numbers in the table because the mistake is part of the story. Your verifier is only as good as your schema. Multi hop reasoning: Sonnet 4.6 78%, GPT 5.5 71%, Mistral 3 8B 51%. Retry didn't help. The model would hallucinate reasoning paths consistently. This is where the capability gap was real. Creative summarization: Sonnet 4.6 4.2 out of 5, GPT 5.5 3.9 out of 5, Mistral 3 8B 3.1 out of 5. Expected. Interpretation: high verifiability tasks seem simpler in the sense that weaker model plus verifier can approach frontier performance. Low verifiability tasks show the expected gap. Limitations: n=120 is tiny. Need 10x for confidence. Our verifier is just JSON Schema plus regexes. Constrained decoding might change the calculus entirely. I also didn't control for prompt length well. Any prompt over 8k tokens was excluded because Mistral 3 8B degrades near its limit, which probably skewed the sample. submitted by /u/DragonfruitAlone4497 [link] [comments]

Fable 5 feels impressive on paper, but worse than Opus for consulting work so far...

Today I decided to run a practical test with Fable 5. I gave the same prompt to Claude Opus 3.8 and to Fable 5 with reasoning set to high. The task was very close to the kind of work I do in consulting: take an existing PowerPoint deck, integrate feedback from a meeting transcript, and create a new version while respecting some strict instructions. The key instruction was simple: do not touch the previous slides. If a slide needs to be modified, duplicate it and make the changes on the new version. At first, I was really confident about Fable. Its initial answer and roadmap were excellent. It understood the task well, structured the work properly, and did not make any obvious mistakes. Opus, on the other hand, made one small mistake early on, but I did not even need to intervene much because it corrected itself later in the process. But the final result was clearly better with Opus. Fable felt surprisingly lazy in execution. It did not do everything it said it would do, made changes to slides it was not supposed to touch, and ignored the instruction to duplicate slides before modifying them. I had to insist several times and re-prompt much more than I usually need to with Opus just to get something decent. The most surprising part was that all the image icons in the PowerPoint generated on the Fable side simply disappeared. When I asked about it, Fable told me it did not have the right version to work properly on PowerPoint files. Meanwhile, with the exact same task on the Opus side, there was no issue at all. The images were integrated correctly, and the deck remained usable. That difference alone was pretty crazy. Opus was not perfect, but it followed the operational constraints much better, preserved the structure of the deck, handled the images properly, and produced a more usable consulting deliverable with less friction. I understand that Fable may be extremely strong on benchmarks, programming, scientific tasks, or i don't know. But for my use case, which is consulting work involving PowerPoint, meeting transcripts, structured feedback, version control, visuals skills, and strict client-deliverable instructions, Fable currently feels like a regression compared with Opus. Curious to know if others are seeing the same thing: is Fable actually better for your real-world knowledge work, or mostly better on paper and coding benchmarks? submitted by /u/BenShutterbug [link] [comments]

Deepseek said it's chatgpt https://www.reddit.com/r/OpenAI/comments/1u1ytjm/wtf_is_this_deepseek_saying_its_chatgpt/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=1&utm_content=share_button

submitted by /u/Prior-Reputation3018 [link] [comments]

Introducing Papers Without Code [P]

Hi, Niels here from the open-source team at Hugging Face. I've recently relaunched paperswithcode.co as a source for finding the state of the art (SOTA) across various AI domains, from 3D generation to AI agents. This is done by automatically parsing research papers published on arXiv/Hugging Face, enabling leaderboards to be created. See BrowseComp below as an example (a scatter plot and a table are available for each benchmark). - Scatter plot (you can hover over the dots to see the models): https://preview.redd.it/9rz2r3ffcf6h1.png?width=2880&format=png&auto=webp&s=b3f8e7a870802f6ef8227ecc0619e9e1057554b0 - Table: https://preview.redd.it/qoqriddw5f6h1.png?width=2862&format=png&auto=webp&s=a0034574f693847537037013672fb61daf27b16e As you can see, I've added support for viewing evals for closed-source models, too, given that many benchmarks are nowadays dominated by them, like GPT-5.5 and Mythos 5. You can always disable viewing closed-source evals with a toggle or in your PwC settings: https://preview.redd.it/p3k6jt6q6f6h1.png?width=1582&format=png&auto=webp&s=40149e51d6b326a77e53e33baf70d9850b3de365 When you turn them off, here's what the open model leaderboard looks like: https://preview.redd.it/tg42sin36f6h1.png?width=2838&format=png&auto=webp&s=1330a117ae9b4e0ce6d459493ae9e8f64107310a Closed-source papers are treated as regular "papers", although they can be any source, like a blog post (given that PwC supports submitting any source beyond arXiv). See the GPT-5.5 or Mythos 5 papers as examples, with their evals at the bottom. Notice the "closed" tag on their evals. Hence, you could jokingly call these "papers without code". Let me know what you think of this, and whether anything needs to be changed or added! Kind regards, Niels submitted by /u/NielsRogge [link] [comments]

The Great Reframing...

I have an economic theory about Anthropic's recent blog post "When AI Builds Itself," in which they requested: "We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology." What I'm questioning is whether this is genuine goodwill or a smokescreen for a technical failure driven by data poisoning, diminishing returns, and public market economics. Correcting the Scaling Law Assumptions Early scaling hypotheses (Kaplan et al., 2020) suggested throwing compute almost entirely at model size. The modern compute-optimal scaling law, formalized by DeepMind's Hoffmann et al. (2022) in the "Chinchilla" paper, corrected this: L(N, D) = (A / N^α) + (B / D^β) + E You cannot just scale parameters (N); you must scale the dataset (D) in roughly equal proportion. But there is a hidden trap: the term E. This represents irreducible error, the inherent entropy of text. As parameters and data approach infinity, loss asymptotes at E rather than dropping to zero. An eventual plateau is mathematically baked in. The critical economic question is whether we are hitting that asymptote now. The Data Poisoning Problem The Chinchilla law assumes dataset D is high-quality, human-generated text. That assumption is breaking down. The internet is now heavily polluted with LLM-produced content, and when models train recursively on synthetic output from other models, they suffer from Model Collapse (Shumailov et al., 2023). The tails of the data distribution disappear, model understanding degrades, and error rates climb. This provides a clear catalyst for the inverse scaling documented by McKenzie et al. (2023), where more poisoned data fed into larger models actually worsens complex reasoning. Capabilities Follow S-Curves Even if cross-entropy loss continues dropping slowly, economic capabilities (passing the bar exam, writing reliable code) do not scale linearly with it. As Schaeffer et al. (2023) showed, emergent abilities follow sigmoidal S-curves. A model hits a loss threshold, unlocks a capability, and performance then flattens at the top of the curve. Spending ten times the compute to squeeze out the next 0.01 drop in loss may yield zero new monetizable capabilities. The Mythos Black Box Anthropic has released no technical details about Claude Mythos: no parameter count, no training token count, no compute figures. There is open speculation that Mythos is among the largest models ever trained, possibly the largest, with a token count to match. If true, Anthropic may have run the most expensive experiment in AI history and hit the data poisoning wall harder than anyone. At that scale you cannot quietly retrain while telling investors everything is on track. The pause request reframes this cleanly: rather than disclosing that the largest training run ever attempted may have underperformed, or that the next run requires solving a fundamental data quality problem first, you shift the narrative to safety and societal readiness. The timing and the financial incentives make that reframing at minimum convenient, and at maximum deliberate. The IPO and the Euphemism Anthropic recently submitted a confidential draft S-1 to the SEC. If you are heading into a highly anticipated IPO, how do you explain to Wall Street that compute-optimal scaling is hitting a wall? How do you justify hundred-billion-dollar data center CapEx if your dataset is poisoned and your capability curve has flattened? You reframe it. Anthropic's writing on Recursive Self-Improvement warns of a near-future where AI models rapidly accelerate their own development, requiring a pause for societal safety. If my theory holds, they are recasting a mundane engineering plateau as an optimistic near-apocalypse. Rather than telling public markets "we are running out of pristine human data," they say "we are dangerously close to a runaway intelligence explosion." A call to pause becomes a financial strategy: slow unsustainable cash burn, prevent open-source competitors from catching up while the synthetic data problem gets solved, and protect valuation heading into an IPO roadshow. They are not pausing because AI is becoming dangerous. They are pausing because the current paradigm is running out of gas. Note: This is speculative economic and technical analysis and does not constitute financial advice. Sources Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models (Chinchilla): https://arxiv.org/abs/2203.15556 Kaplan et al. (2020) — Scaling Laws for Neural Language Models: https://arxiv.org/abs/2001.08361 Schaeffer et al. (2023) — Are Emergent Abilities of Large Language Models a Mirage: https://arxiv.org/abs/2304.15004 Shumailov et al. (2023) — The Curse of Recursion / Model Collapse: https://arxiv.org/abs/2305.17493 McKenzie et al. (2023) — Inverse Scaling: When Bigger Isn't Better: https://arxiv.

Anthropic just released Claude Fable 5 a Mythos-class model for general use, with safety classifiers that fall back to Opus 4.8 on ~5% of sessions

Anthropic dropped two models today: Claude Fable 5 (general availability) and Claude Mythos 5 (restricted to cyberdefense partners). The short version: Fable 5 is their most capable model ever released publicly, and they’re being unusually transparent about how they’re handling the risks. What’s actually impressive: -Stripe compressed months of engineering into days with it. In a 50-million-line Ruby codebase, Fable 5 did a codebase-wide migration in a day that would have taken a full team 2+ months by hand.  -On vision tasks, it beat Pokémon FireRed using only raw game screenshots with no maps or navigation aids. Previous Claude models needed complex helper harnesses to even play it.  -Mythos 5 autonomously conducted novel genomics research over a week, assembling single-cell data for millions of cells across 138 animal species. Its trained model outperformed a recent paper published in Science despite being 100x smaller.  -On Cognition’s FrontierCode eval (production-quality coding), Fable 5 scores highest among frontier models, even at medium effort.  The safety approach is interesting: Rather than just refusing dangerous requests, Fable 5 uses classifiers that silently fall back to Opus 4.8 on queries related to cybersecurity, biology/chemistry, and distillation. Users are informed when this happens, and it triggers in less than 5% of sessions on average.  They ran a bug bounty that produced zero universal jailbreaks in 1,000+ hours of testing. UK AISI made some progress toward one in a short initial window, but no full break.  Pricing: $10/M input tokens, $50/M output tokens less than half the price of Mythos Preview.  Caveat on Pro/Max/Team plans: Free access lasts through June 22, then requires usage credits. They say they’ll restore it as a standard plan feature when capacity allows.  The biology capabilities are wild Mythos-class models outperforming dedicated protein language models on AAV design tasks without being trained for it is a real signal of how much general reasoning ability has jumped. submitted by /u/Direct-Attention8597 [link] [comments]

The AI productivity paradox that needs to be addressed rn

The conversation around AI coding is still stuck on velocity and its completely missing the real operational bottleneck -> DEBUGGING I use a combination of tools like GitHub Copilot, Cursor, and generic agentic code gen tools(whichever give me the most credits that week) , dropping a 300-line functional block from a natural language prompt takes about a minute. On paper, developer velocity should have been increased by 69 times. but i feel like the bottleneck hasn't disappeared; it just shifted down the pipeline. Like i traded manual work for incredibly frustrating debugging. LLM code looks fine on surface but like when u go through line to line, you feel like its built on sand i mean sure if it works it works but like one thing i struggle with is ghost features, like if i accidentally suggest a feature then the LLM is gonna shove it in my code, even if i say no later on. (if someone knows how to fix do dm) idk about ya'll but i'd much rather have a ai llm that takes like 1 hour to write 500 lines of code if that means i have to debug less. another thing how are you handling validation boundaries? are u using runtime timeout scripts or smth open source like gitagent? also this is gonna sound weird but i kinda have trust issues when a llm spits like 300-400 lines in under a minute (idk why) sorry for my bad english, im not a native speaker submitted by /u/SpicyTofu_29 [link] [comments]