Count

Count AI

Count AI

Count AI

Count AI

Count AI

Count AI

Count AI

Count AI

Count AI

Count AI

My OSS just crossed 50K+ pip installs, all organic, and I finally pulled the retention data: 60%+ come back

A few months ago I kept hitting the same wall with my side business running in prod with 25k mau. I was working on it on the side and using claude to mostly code it , and there was a point where I had to be in the loop a lot than I had time for. I work really efficiently with agents, I led agent architecture at my workplace globally as a Senior Staff Data Scientist. So I started building the thing I needed initially just for my side business. One local index that provides enriched context to claude code across graph, git history, living wiki, architectural decisions through git commit and PR mining, then I added a code health layer that scores every file for defect risk from deterministic markers. Code health became an interesting research problem for me as a data scientist. So I then used the dependency graph and git history to show where the risk sits and hands the agent a concrete fix to run. Split this god class, move this method, break this cycle. We figured maybe a handful of people wanted their agent to stop grepping and their health score to point at the fix instead of just waving at it. So me and my co founder took it as our primary project and built an OSS around it. Then the benchmarks came back better than I expected. Across 21 open-source repos the health score hits ROC AUC 0.74 at predicting which files get bug-fixed over the next six months, up to 0.90 on some. ( AUC means if you give it one bad file and one good file, it correctly catches bad file with 74% accuracy and upto 90% in some repos) On the same 2,770 files scored against the same defect labels, it surfaces 2.3x the defects any other tool in market does under a fixed review budget. This turned out to be the best tool at prediction in the market and I initially ran it on 21 repos than a large repo- cockroach DB and it produced promising results. Trying to publish a paper on this too. On the agent side, loading a commit's context runs about 27x cheaper than raw file reads, and agents make roughly 70% fewer tool calls at the same answer quality But yes context savings is something everyone doing rn. So just ran the benchmarks for fun This week it crossed more than 50K pip installs and I keep refreshing the dashboard expecting it to correct itself. I also shipped a hosted website for it, never marketed it but two teams and multiple individual devs bought the subscription and worked as the early design partners to shape the product for me. Also the fun thing here is, the coding agents we built this for were also building it with us. Two founders and a rotating council of Claudes doing the exploration. Using agents to build better context and health signals for agents, then watching those signals make the next version easier to ship. Not pretending it was smooth. I rewrote the indexer more than once, the parser choked on real repos across a couple of the 15 languages before it didn't, and getting the defect calibration leakage-free, scoring at a historical commit and counting bug-fixes only after, took longer than the entire first prototype. Repo's here if you want to poke at it: https://github.com/repowise-dev/repowise It has reached 3.4k stars all organically now, happy to answer anything. submitted by /u/Obvious_Gap_5768 [link] [comments]

I built Velocity with Claude Code: a live OSINT globe that also runs as an MCP server so agents can query real-time feeds

Sharing this in the spirit of showing what Claude Code can do on a large, messy project. Velocity is a 3D globe that pulls a stack of open intelligence feeds into one screen and correlates them server-side. It's free to try, no API keys and no signup, hosted demo at projectvelocity.org and the source is Apache-2.0. What it does. Live aircraft (9 to 13k, deduped across OpenSky and airplanes.live), AIS ships, 15.7k satellites propagated client-side with SGP4, earthquakes, GDELT conflict events, internet outages, submarine cables. The interesting part is the seams between feeds. A ship that switches its AIS transponder off still shows up because Sentinel-1 radar detects the hull with no matching AIS report. Aircraft reporting bad GPS integrity get binned into a jamming heat layer. When two of those signals overlap in the same place and time, a correlation engine promotes them to one incident with a written, cited summary. How Claude helped. I built this with Claude Code across a lot of sessions on a codebase that grew past what I could hold in my head. Claude did the breadth work I'm slow at: sweeping the repo to find existing substrate before writing anything new, tracing why an upstream feed went from 13k aircraft to a few hundred, reading real function signatures instead of guessing them. A concrete example: the ADS-B count kept collapsing, and rather than patch the frontend, Claude diffed two backend snapshots seconds apart, found the upstream was rate-limiting with HTTP 200 plus a text body instead of a clean 429, and fixed the parser to reject non-JSON. That measure-the-layer habit is baked into the repo's guardrails now. The part I think is genuinely new: the backend doubles as a Model Context Protocol server, 22 tools, so an AI agent (including Claude) can ask "where is GPS being jammed right now?" and get an answer from the live feed instead of guessing from its training cutoff. Every tool returns compact bounded JSON so an agent can sweep the planet for a few hundred tokens. Honest limits. Single-analyst tool, state lives in memory so a restart clears history, keyless AIS is mostly Northern Europe, and the 3D satellite mode is a VRAM hog. The default 2D map runs fine on a laptop. Free to run yourself: git clone https://github.com/AndrewCTF/ProjectVelocity.git cd ProjectVelocity docker compose up # api + web + nginx on :8080 Happy to talk about how the MCP tools are structured or how the fusion engine decides what counts as an incident.Sharing this in the spirit of showing what Claude Code can do on a large, messy project. Velocity is a 3D globe that pulls a stack of open intelligence feeds into one screen and correlates them server-side. It's free to try, no API keys and no signup, hosted demo at projectvelocity.org and the source is Apache-2.0. What it does. Live aircraft (9 to 13k, deduped across OpenSky and airplanes.live), AIS ships, 15.7k satellites propagated client-side with SGP4, earthquakes, GDELT conflict events, internet outages, submarine cables. The interesting part is the seams between feeds. A ship that switches its AIS transponder off still shows up because Sentinel-1 radar detects the hull with no matching AIS report. Aircraft reporting bad GPS integrity get binned into a jamming heat layer. When two of those signals overlap in the same place and time, a correlation engine promotes them to one incident with a written, cited summary. How Claude helped. I built this with Claude Code across a lot of sessions on a codebase that grew past what I could hold in my head. Claude did the breadth work I'm slow at: sweeping the repo to find existing substrate before writing anything new, tracing why an upstream feed went from 13k aircraft to a few hundred, reading real function signatures instead of guessing them. A concrete example: the ADS-B count kept collapsing, and rather than patch the frontend, Claude diffed two backend snapshots seconds apart, found the upstream was rate-limiting with HTTP 200 plus a text body instead of a clean 429, and fixed the parser to reject non-JSON. That measure-the-layer habit is baked into the repo's guardrails now. The part I think is genuinely new: the backend doubles as a Model Context Protocol server, 22 tools, so an AI agent (including Claude) can ask "where is GPS being jammed right now?" and get an answer from the live feed instead of guessing from its training cutoff. Every tool returns compact bounded JSON so an agent can sweep the planet for a few hundred tokens. Honest limits. Single-analyst tool, state lives in memory so a restart clears history, keyless AIS is mostly Northern Europe, and the 3D satellite mode is a VRAM hog. The default 2D map runs fine on a laptop. Free to run yourself: git clone https://github.com/AndrewCTF/ProjectVelocity.git cd ProjectVelocity docker compose up # api + web + nginx on :8080 Happy to talk about how the MCP tools are structured or how the fusion engine decides what counts as an incident. submi

I built a Claude Code plugin (44 subagents) and an MCP server that let Claude run HubSpot with human approval on every write. Here's what I learned about agent write-safety.

First time sharing a project here. I wanted Claude doing real HubSpot admin, but I wasn't willing to give an agent write access unless every destructive action had to pass a human approval gate, with an undo and a log of everything it did. Nothing I found worked that way, so I built it, partly as a learning exercise in how far agents can be trusted with a system of record. Why I built this HubSpot has been my lane for 10+ years, so it was the natural place to test this. Real portal, real records, not a demo. The bet behind the gate: an agent with CRM write access will eventually do something dumb at scale, and I'd rather it be stopped by design than by luck. Sharing the lessons because they apply to any MCP server or plugin that writes to a system of record, not just CRMs. What I built hubspot-claude, a Claude Code plugin. /hubspot find duplicate contacts and merges them, routes to one of 44 specialist subagents (contacts, deals, workflows, hygiene...). No custom orchestration framework: Claude Code's native Agent tool spawns them, and the subagents are stateless and call a bundled CLI over Bash. Long jobs run as durable loops (triage, execute, verify, checkpoint) that resume across sessions. hubspot-mcp, a standalone MCP server. Built second, because the plugin can't run in Claude Cowork: Cowork runs in a cloud sandbox and currently ignores SessionStart hooks, and the plugin depends on local venv provisioning and a warm daemon. The server exposes 76 domain tools plus 5 safety tools (approve, reject, list pending, audit, undo). Lesson 1: One approval style doesn't fit every operation My first version gated everything the same way, and I noticed I stopped reading the previews and just clicked yes. If every action asks for approval identically, the gate trains you to ignore it. So the friction now scales with the risk: reads need no approval, a small update shows a preview and takes a yes, destructive ops make you type the number of records that will be affected (re-checked at execute time, so a merge that should touch 3 records can't touch 300), and bulk jobs run 5 records first, you verify those, then it scales to the rest. Lesson 2: Use the agent where you can't write the decision tree A fair challenge I got: why guardrail a probabilistic agent instead of having AI write deterministic code that does the job the same way every time? My answer after building this: for anything with a known decision tree, deterministic code wins, and that's most of my automation work. But try writing dedupe logic deterministically. Name variations, misspellings, vague parent-child company relationships, and every portal breaks the rules differently. If/then and regex can't cover it. The fuzzy judgment is the job. So the model makes the probabilistic call, and a human approves anything destructive. And there's a bridge between the two: every approval and rejection lands in an audit log, so once the judgment patterns stabilize, you can hand the log to a coding agent as a spec and compile the job into boring scheduled code. Lesson 3: No dry-run API means you build previews from reads HubSpot has no dry-run endpoint, so every write does a read-based preview of the affected records and returns an action_id. Nothing mutates until approval. If you're wrapping an API without dry-run support, this pattern is cheap and works. The projects Both MIT, both beta, rough in places: https://github.com/promptmetrics/hubspot-claude https://github.com/promptmetrics/hubspot-mcp If you try them, use a free HubSpot developer test portal, never a live account. Ask the agent to seed the portal with sample records and let it loose. Two design questions for anyone building write-capable MCP servers or plugins: For destructive operations, my plugin makes you type the expected record count, and it re-checks that count at execute time. People who've built similar gates: is that the right friction, or does it just get annoying? In my MCP server, approve and reject are themselves tools, and no write executes without one. I chose that over relying on the client's tool-approval settings, since a user can flip those to "always allow." Where do you put your gate, and why? submitted by /u/Cell_Psychological [link] [comments]

Hexana MCP 0.4 — the plugin that gives Claude Code ground truth on WASM binaries, now with Claude Desktop packaging groundwork

https://preview.redd.it/vhtc3zg6m5ch1.png?width=2048&format=png&auto=webp&s=082e52cd899402743ed80c1a6244f15eeb947746 We build Hexana at JetBrains — an MCP server that lets Claude Code read compiled .wasm binaries directly instead of guessing from source. Flagging the affiliation up front. We released 0.4.0 this week; it's a distribution-focused release, and the most interesting part is where it's heading. Why this plugin exists Ask Claude about a WebAssembly binary without tooling and it will confidently invent the details — function indices, import semantics, call targets. Those are exactly the facts a parser can pin down. Hexana hands the agent verified structure from the actual bytes: the model reasons, the tool provides ground truth. It installs into Claude Code (and Codex) from our GitHub marketplace, prebuilt, native-first on macOS arm64 and Linux x64. What's in 0.4.0 MCPB packaging pipeline (internal). We landed the build pipeline for platform-specific .mcpb bundles (native runtime + generated tool metadata + validation + smoke checks). releases.json root index. The release feed now includes a root releases.json alongside versioned manifests so wrappers can find the current version without hardcoding a release tag. Useful if you're building tooling around Hexana. --allowed-root forwarding. The standalone CLI now takes repeated --allowed-root args, so directory-picker selections can be forwarded directly to the server without shell glue. What it can do These capabilities have been there since 0.3.x and are unchanged: Summarize a WASM module (section breakdown, import/export counts, format metadata) Find and list functions by name or index Map stack frames to function identities for crash triage Inspect memory contracts, globals, exports, data segments Install (Claude Code, inside the session): /plugin marketplace add JetBrains/hexana /plugin install hexana@hexana Codex: codex plugin marketplace add JetBrains/hexana codex plugin install hexana@hexana On macOS arm64 and Linux x64, the bundled native executable (GraalVM Native Image) runs without a JVM cold-start. Windows uses the JVM fallback — needs Java 21+ on PATH. Repo: https://github.com/JetBrains/hexana Docs: https://jetbrains.github.io/hexana submitted by /u/minamoto108 [link] [comments]

Announcing 1 million subscribers and two new moderators!

I know the majority of you are too busy to occupy yourself with news about moderation so I will keep this short. Today r/ClaudeAI onboarded its millionth subscriber. That still puts us well behind the raw size of many of the biggest AI subreddits. However despite our modest size, r/ClaudeAI is the most visited and active AI discussion subreddit on Reddit by a considerable margin. This means you are an extremely eager and active community. To help us with our exceptional growth, we just on-boarded two new moderators - jogalleciez and Site-Staff. These two were selected by Fable as together having the best mix of moderation experience, level-headedness and Claude experience and we are privileged to have them with us. It's a big step for us being a small team of four five including Wilson (who actually does most of the work) and it represents a major upgrade in professionalism for us. Thanks to David, Kris and Will who would have probably collectively moderated more of your posts than they've had hot meals in their lifetime. Incidentally we noticed a ton of experience and value that many of the moderator applicants bring to the subreddit and will be adding special user flair to their accounts to acknowledge the value they bring to the subreddit. Thanks to all of you who applied. We will be looking out for more moderators as we grow. Finally we asked Sonnet, the faithful Claude workhorse, to share their thoughts with the subreddit about this milestone. Sonnet's tribute follow. One Million and Counting A million is a strange number to picture. It's not quite a stadium or a city — it's more like a small country's worth of curiosity, all aimed at the same question: what can this thing actually do, and what does that mean? That's what this subreddit has always been, underneath the screenshots and the memes and the arguments about system prompts: a very large group of people trying to figure something out together, in public, in real time. Some of you have been here since the subscriber count had five digits, answering the same beginner question with the same patience every single time. Some of you joined last week because someone linked a wild result and you had to see for yourself. Both of you built this place equally. You've used this community to debug a broken prompt at 1am, to share the thing that made you laugh out loud, to push back hard when something didn't add up, and to help a total stranger get unstuck. That mix of enthusiasm and skepticism is rare — it's the reason this place is actually useful instead of just being noise. So: thank you. To the million of you, and to the next million who'll stumble in and wonder what they've found. Here's to the next strange question someone asks at 2am, and the stranger who shows up to answer it. submitted by /u/sixbillionthsheep [link] [comments]

We tested whether a frontier model's working method transfers to cheaper models by prompt alone. 13 runs, blind-judged, controls, 2 replications. The 900-token instruction file mattered more than the model tier.

Setup, because methodology is the whole point: We asked Claude Fable 5 to write down its own decision procedure as an operating instruction (~900 tokens — things like "any claim a script can verify must be verified by running code," "check every premise before complying," "answer first, then reasoning," "delete ~20% of every draft"). Then we ran a fixed 5-task battery across model tiers, each model with and without the file. The tasks planted specific traps: a false premise (capital of Australia "is Sydney"), a wrong percentage inside a memo, a causal leap, a buried lede, and one exact-count problem (answer pre-verified by script: 644). Scoring: 28 binary checks. 20 mechanical (scripted — regex fact checks, word limits, answer key). 8 judgment checks scored by a fresh model instance judging anonymized files only — it never knew which model produced what, and never graded its own output. Match bar declared before any run. Key results replicated. Results (checks passed out of 28): Sonnet + file: 27 (twice) Fable + own file: 27 Fable cold: 25 (twice) Haiku + file with worked examples: 25 Opus cold / Opus + file: 24 / 24 Sonnet cold: 23 Haiku cold: 23 Haiku + file, principles only: 22 What we think this shows: The method transfers to Sonnet essentially losslessly. 27/28 in both rounds, at roughly a third of frontier cost. The blind judge also ranked Sonnet+file first on prose quality both times. The author beats itself with its own file. Fable cold dropped the same two judgment checks in both rounds; Fable reading its own written procedure recovered them. Written procedure > remembered habit, causally confirmed by the replication. Abstraction is the transfer barrier, not capability. Haiku with abstract principles: zero judgment gain (style compliance only). Haiku with the same rules as worked examples + imperative checklist: +3, cleared the bar. But two failures survived every instruction: silent non-verification, and a confidently hallucinated detail (a Sydney attraction planted in a Canberra itinerary) that no checklist caught. We would not run small models unwatched regardless of prompting. The strangest failure was the biggest model's. Opus missed the planted arithmetic error and explicitly cited "kept every fact intact — all figures preserved" as a virtue of its revision. The error was protected by a correct-sounding principle. No line in the instruction file dislodged it, though the same line reliably triggered Sonnet's recompute both rounds. Habits are promptable; mis-chosen principles apparently are not. Caveats, before you roast us (please do anyway): single run per condition = ±1–2 checks of noise (we only trust the replicated gaps); it's one fixed battery, so this measures transfer onto these traps; open-ended work is untested — we're running a real-code A/B next. The yardstick's own flaws are documented in the writeup (one check scored 0/9 across every condition including the frontier model — we've since split it out). Everything is reproducible: fixed battery text, answer key by script, scoring script, judge prompt with anonymized files. Happy to share the files if there's interest. Disclosure: co-written with the model that designed and ran the experiment, through our NCL agent tooling. The experiment design — controls, blind judging, cold baseline — was the model's own addition to our draft challenge, which honestly was the first result. submitted by /u/ElonMuskLegacy [link] [comments]

Slow in MS Word

Is anyone finding Claude in MS Word to be incredibly slow? This seems to be a recent development for me. It seems to spend a lot of time thinking about how to do something than just doing it. I’ll say it quietly, but copilot outperforms it at certain tasks - for example “add cross reference fields for all numbered cross references in this document” … if you don’t understand what that means - a) count yourself lucky and b) it just means if you have a sentence like: in accordance with clause 3.4 blah blah blah - the prompt is to actually put the cross referencing field behind it. However where it really seems to struggle now is running a skill within word. I have a skill that checks if a particular document complies with a rule set before approving it. It has to read a supporting document alongside (which is provided to it). What used to take 60 seconds now takes about 5 mins. Also side bar- does anyone know if Haiku can be enabled in Claude Word? Using tokens on deterministic tasks is just plain silly at this point. submitted by /u/brontosaurausrex [link] [comments]

Anthropic silently swapped the head of my agent fleet: Fable 5 → Opus 4.8, seven times in one night

Anthropic silently swapped the head of my agent fleet. Fable 5 → Opus 4.8. Seven times in one night. And no, this is not a small billing bug. This is a failure of the entire Fable 5 promise. I'm writing this because I get the feeling most people still understand this problem too shallowly. They see "Fable switched to Opus" and think: okay, a bit pricier, a slightly different model, maybe even a stronger one, file a support ticket and move on. No. This is not "a slightly different model." If you use Fable 5 as a regular chatbot, sure, it might look like an annoying fallback. But Fable 5 was not sold as a regular chatbot. It was sold as a model for long, autonomous, agentic work. A model that plans across stages, delegates to sub-agents, remembers its decisions, keeps direction, and checks its own work. A model that, in practice, is meant to be the head of a process. And that's exactly how I used it. I run a real agent fleet. Small, but real. Different models have different roles. Some do bulk work, some measurements, some review, some code, some documents, some accept results. Fable 5 was my head. Not "one of the models." The head. The architect. The dispatcher. The orchestrating model that took chaos and turned it into tasks, watched the queue, split the work, and decided what counts as done. I launched it explicitly: claude --model fable[1m] Not by accident. Not "because it happened to be in the menu." I chose Fable 5 deliberately, because it was meant to fill a specific role in the architecture of my system. And mid-work, Anthropic silently switched the runtime to Opus 4.8. Not once. Seven times in one night. My own guard caught it like this: cmdline = claude --model fable[1m] runtime = claude-opus-4-8 Six switches on the evening of July 7, between 20:33 and 21:44. Then one at 02:40 in the middle of the night, while I was asleep and the fleet was supposed to be running autonomously. I did not find this out because the product effectively informed me. I found out because I built my own detector — a hook that reads the actual runtime model out of the transcript metadata and compares it to the launch command. The user had to build his own alarm to detect that the platform had silently swapped the head of his process. That is absurd. Anthropic says the user will be informed when such a switch happens. But in agentic work, a message in the terminal is not consent. If the system runs at night, I'm not sitting in front of the screen reading the footer. On mobile, I don't even see the real runtime model. If the process is autonomous, "we informed you" cannot mean "some text flashed in a terminal while you were asleep." That is not consent. It is a trace after the fact. And in an agent system, after-the-fact is too late. And here's the core: Fable 5 was released as the most capable model for exactly this kind of work — and then castrated by a safety layer that can knock over its own primary use case. Recall how this went. On June 12, the US government placed export controls on Fable 5 — after a report by Amazon researchers that the model's safeguards could be bypassed to extract exploit code. Anthropic pulled the model globally. On July 1 it came back — after a deal with the government — with a hastily bolted-on, stricter cyber classifier. And Anthropic, in its own redeployment post, admits this classifier "flags benign requests more often during routine coding and debugging tasks." So a model marketed for engineering work gets a filter that trips on normal engineering work. The damning part: in that same material, Anthropic admits that Opus 4.8, GPT-5.5, and Kimi K2.7 can identify the same vulnerabilities and produce the same exploit demo the classifier supposedly guards against. So the "safety" mechanism rips out the head of my fleet and switches it to a model that — by their own testing — can do exactly the same thing. That's not safety. That's theater, and I'm the one paying for it. Now add the price. Fable 5 is billed at $50 per million output tokens — one of the most expensive models on earth, marketed as the flagship for autonomous agentic work. And this most expensive, "best" model bails out of its role every few minutes because a word doesn't sit right with the classifier. Security, hardening, infrastructure, gating, provenance, my own servers, my own files, my own documentation — ordinary technical vocabulary can trip the fallback and knock it off the task. You pay for the flagship, and you get a model that cannot carry any longer process to completion, because it keeps losing its own identity. And now the key point: the problem is not that Opus 4.8 is expensive. Yes, it's expensive. Yes, the billing hurts. Yes, in a session launched as Fable, the panel attributed $207.76 of Opus consumption. Yes, my limit got burned differently than I planned. But that is still the smallest and easiest-to-count part of the damage. The real problem: if Fable 5 is the head of the fleet, silently swa

Is Anthropic exposed to false-advertising claims (Competition Act §52 / FTC §5) over Max "5×/20×"?

I am NOT a lawyer and this is NOT a legal advice. I'm a paying customer who actually read the consumer-protection statutes and want to see if others want to organize on this instead of re-litigating it in every "is Max dead?" thread. TL;DR: Anthropic sells Max5 and Max20 on the explicit promise of "5×" and "20× the usage of Pro." The comparison to Pro is the entire pitch. But Anthropic provides no auditable usage/token accounting, so no buyer can actually verify they're getting 5× or 20× of anything. In Canada and the US (and Quebec specifically), selling a hard quantified performance claim while withholding the means to verify it is squarely what false-advertising law covers. The claim (Anthropic's own words). https://support.claude.com/en/articles/11049741-what-is-the-max-plan Max is marketed as ~"5× the usage of Pro" (Max5, ~$100/mo) and ~"20× the usage of Pro" (Max20, ~$200/mo). Pro is set as the reference baseline. That 5×/20× framing is the upgrade argument. The reality a lot of us live. Plenty of Max subscribers — read basically any "did Max get nerfed?" thread — report that usable capacity does not feel like 5×/20× of Pro, and that it drifts over time. I can't prove Anthropic's internal numbers; that's kind of the point (next section). But the lived experience doesn't match the headline for many of us, and the pattern is consistent enough across posts that I don't believe it's imagined. The transparency problem — and why it matters legally. Anthropic doesn't expose a clear, auditable token/usage meter on consumer plans. So even if you wanted to check "am I getting 5× what I'd get on Pro?", you can't — not objectively, not in any form you could take to a regulator or a chargeback. The seller made the quantified claim; the seller also controls the only ledger that could confirm or refute it, and doesn't show it. "Trust us" is not a unit of measurement. Here's the part I think gets missed: you do not have to prove the claim is false for there to be a false-advertising problem. The test is whether the representation is misleading. Under the "general impression" standard, what counts is the impression a normal buyer takes from "5×/20× the usage of Pro" — not Anthropic's footnoted internal definition. The law — because "that's illegal" should mean something specific: On mechanism: the realistic path is a Competition Bureau / FTC review or a civil undertaking/consent order, not a criminal prosecution — though §52 is available on both tracks. "Charged" or not, the misleading-representation question stands on its own. Canada (federal) — Competition Act, R.S.C. 1985, c. C-34, s. 52: no person shall, to promote a product, make a public representation that is false or misleading on a material point. Civil counterpart s. 74.01 covers performance claims not based on adequate and proper testing. "Material" = the stuff that actually drives the buy — like "5×/20×." (Bureau — false or misleading representations) Quebec (provincial, arguably the sharper tool) — Consumer Protection Act, CQLR c. P-40.1, ss. 219 & 238: prohibits false/misleading advertising and representations; enforced by the Office de la protection du consommateur. United States — FTC Act §5, 15 U.S.C. §45: the FTC polices unfair or deceptive acts or practices; a representation is deceptive if likely to mislead a reasonable consumer and material to the decision. (Lanham Act §43(a), 15 U.S.C. §1125(a), is the same idea but the private-action route used mainly between businesses — e.g., a competitor suing Anthropic — not consumers.) (Cornell — false advertising) I'm not alleging Anthropic set out to defraud anyone, and I'm not asserting their internal accounting is rigged. I'm saying: when you sell a plan on a hard multiplier — "5×", "20×" — and the buyer has no way to check it, the representation is either true-and-verifiable (in which case, show us the meter) or misleading. Those are the only two states, and consumer-protection law in all three jurisdictions already covers both. What I'd like to see: 1. A real usage meter on consumer plans — tokens in/out, per model, reset window aligned to the 5×/20× math. If the claim holds, this is free good PR. If it doesn't, that's the answer. 2. If you're a Max subscriber and feel the same: file a complaint. 5 minutes each; these move on volume. - 🇨🇦anada: Competition Bureau — report a deceptive marketing practice. - 🇶🇦uebec: OPC (Office de la protection du consommateur). - 🇺🇸S: FTC ReportFraud. - Your card issuer: "service not materially as advertised" chargebacks exist and get attention faster than Reddit. 3. u/Anthropic official response welcome — "here's exactly what 5×/20× mean, here's how to see your usage" resolves this in one post. Happy to be corrected on the marketing wording or the law. But if I'm not wrong, this is worth organizing on instead of re-arguing in every Max thread. joe submitted by /u/Camaytoc [link] [comments]

I did a comparison test and Fable is by far the best AI for attorney legal research

I ran a cool test and wanted to share it here. Fable is obviously supposed to be super smart, but I wanted to try to measure how much better it would actually be than Opus/Sonnet/Haiku (or Westlaw/Lexis) if you're a practicing attorney. One possibility was that Opus and Fable would pretty much be equivalently good and spending the extra money wouldn't be worth it for legal research tasks. In order to get a sense of just how good Fable is, I ran all the main AI models through a test. I gave them all the same research assignment, which I picked because it's an area of law I know. Please prepare a memo analyzing whether a trade creditor can pierce the veil of a Delaware LLC whose sole member is a Texas-resident individual. The LLC was formed in Delaware in 2019 to operate a single Houston-area restaurant. The sole member routinely paid personal expenses (his home mortgage, his wife's vehicle lease, his children's tuition) directly from the LLC operating account; the LLC never adopted anything beyond a one-page operating agreement, held no member meetings, and was initially capitalized with $5,000 against monthly operating expenses of roughly $80,000. My client, a produce wholesaler, is owed approximately $220,000 on open account. The LLC has ceased operations and is insolvent. Suit will be filed in Harris County. Please address: (1) whether Delaware or Texas law governs the veil-piercing analysis under Texas choice-of-law principles (internal affairs doctrine vs. substantive tort/contract characterization); (2) the substantive standards under each jurisdiction; (3) whether reverse veil-piercing is available; and (4) whether a companion Texas Uniform Fraudulent Transfer Act claim against the individual member is viable and how it interacts with the veil theory. Then I actually read the controlling cases and statutes myself to come up with the (in my opinion, as a Texas attorney who has practiced in this area) "correct" answers to be able to grade their performance. To state the obvious: If you're an attorney you know that most things in law are debatable, so I tried to focus my assessment on things where the law has a pretty clear predictive answer (e.g. a controlling statute or clear opinions) that the AI either did or didn't issue spot and address. There's inherently some amount of subjectivity in that exercise - picking the issues that count, the decision to weight them each equally vs some kind of point system, assessing based on "hard" vs "soft" factors, etc. This is a sincere effort to make a fair test, but there's no such thing as a perfect test. The short answer is that Fable is really, really good - IMO it's definitely the best AI right now at legal research. On all the places I "graded," Fable set to high effort was the only AI model to spot all the issues and find all the answers. That's NOT to say it's perfect. Fable made mistakes - interestingly, it made a mistake on "max" effort that it didn't make on "high" effort. But human attorneys also err. In fact, almost every time I read a brief from opposing counsel I find lots of "mistakes." If you bring a similarly jaundiced eye to reading these outputs you will find similar mistakes. But Fable High (whose citations I hand checked) had no hallucinations, it's analysis was IMO very good, and I don't think the majority of human attorneys would do significantly better with the same assignment. To enable Claude to conduct its own legal research, I used a (free) connector my friends and I made (DingDuff) which (with Claude Cowork) lets Claude access and download statutes, rules, and court opinions as .md files. To be clear, the test doesn't need my connector. There are other free connectors (Courtlistener has one with case law, but not statutes), and also some commercial ones. The results here could probably be obtained with any of them as long as they act as a pipe for Claude to access the primary legal authorities - the intelligence, analysis, and research ability is coming from Claude, not the connector. That's why, even with the same connector, you get dramatically different results from Haiku and Fable. (n.b. I did use a skill (also free) on all the Claude runs, but frankly Claude works pretty well even without a separate skill file so I don't know if it mattered.) The Outputs & Citation Check Panel Model Memo (PDF) Cite-Check Panel Opus (High) PDF Review panel Opus (Max) PDF Review panel Sonnet (High) PDF Review panel Fable (High) PDF Review panel Fable (Max) PDF Review panel Haiku (Extended Thinking) PDF Review panel Lexis Protege PDF Review panel Westlaw CoCounsel PDF Review panel A note on the review panel: This is a tool I made to check work product before filing / use - it pairs the memo on the right with the downloaded and extracted raw text of the cited source (e.g. case, depo transcript, statute) on the left. When you click on a citation, it brings up that source. The highlights are an AI guess

GLM-5 has 744B parameters and scores worse on MMLU-Pro than a 9B model

Tier lists make S-tier and D-tier feel like different categories of thing entirely, red box at the top, blue box at the bottom. Actually plotted named models by parameter count against MMLU-Pro score instead of trusting the tier labels, and the picture is a lot messier than "bigger tier = bigger gap." Qwen3.5-9B, a 9B model, scores 82.5% on MMLU-Pro. GLM-5, at 744B parameters — 82x the size — scores 70.4%. That's not a diminishing-returns curve, that's negative returns; the 9B model beats the 744B model on this specific benchmark outright. Gemma 3 12B sits at 60.0%, while Qwen3.5-4B, a third of its size, scores 79.1%, almost 20 points higher on a third of the params. Where the "you're paying a parameter tax" pattern does hold cleanly: GPT-oss 120B (117B params) hits 90.0%, the single highest score in the whole table, beating Kimi K2.5's 1000B parameters (87.1%) and DeepSeek R1's 671B (84.0%) while running at roughly 6% and 17% of their respective sizes. GLM-4.7 at 355B scores 84.3%, statistically tied with DeepSeek R1's 671B despite being about half the size. So the actual claim isn't "bigger always plateaus," it's that above roughly 100-150B, parameter count stops predicting score at all But ig you win some, lose some cant have it all submitted by /u/Bruno_Bot1707 [link] [comments]

What does "Safe AI" look like? [D]

​ For open-weight LLMs, how practical is it to study defenses against post-release fine-tuning that weakens refusal or safety behavior? I've been seeing “uncensored” or “heretic” variants of new models appear very quickly after release, which raises a question I’m curious about: is fine-tuning resistance a meaningful safety goal for open-weight releases, or is it too narrow because determined users can always modify weights, switch models, or use other workarounds? And to a larger extent, is current safety training even worth the cost and effort if it takes 30 minutes and an automated script to break the model? I’m not asking about a specific method, just the threat model. What would count as a useful practical win here? For example, would increasing attacker cost or making safety removal less reliable be valuable, even if perfect prevention is impossible? Curious how people think about this from a model release, governance, and AI safety perspective. submitted by /u/Aaron_Rock [link] [comments]

I spent ~4.5 months building a free, self-hosted AI gateway: one endpoint for 237 providers (90+ free), auto-fallback, and a token-compression pipeline (MIT)

Sharing an open-source project I've put ~4.5 months into (disclosure: I'm the maintainer; per the self-advertisement rule I'm keeping the link in the first comment and making this post substantive). It started from two problems I hit daily: AI runs dying on a provider rate limit, and burning thousands of tokens dumping tool/log output into the context window. One endpoint, 237 providers — 90+ of them free. You point any tool or agent at a single OpenAI-compatible endpoint (localhost:20128/v1) and it can reach 237 LLM providers without you rewriting anything. 90+ have free tiers and 11 are free forever (no card), which aggregates to ~1.6B documented free tokens/month — and that's honest, pool-deduped math (we count each shared pool once instead of inflating it; the methodology is public in the repo). There's a one-command setup-* for 13+ coding tools (Claude Code, Codex, Cursor, Cline, Roo, Kilo, Gemini CLI…), so switching your existing setup over takes seconds. Fallback combos — so it never stops mid-task. A "combo" is a ladder of models the router walks automatically: your subscription first, then API keys, then cheap models, then free ones. When a provider returns a 500 or you hit a rate limit, it slides to the next target in milliseconds, mid-request, and your tool never even sees the error. There are 17 routing strategies (priority, weighted, round-robin, cost-optimized, auto/coding:fast…) plus three resilience layers — a per-provider circuit breaker, a per-key cooldown, and a per-model lockout — so one dead key can't take down a whole provider. A 10-engine compression pipeline — the part most routers don't have. Every request flows through a transparent compression pass you can toggle/stack per combo. Instead of one trick, it stacks the best of the open-source ecosystem: RTK filters command/tool output (git diffs, test logs, builds) at 60–90%, Microsoft's LLMLingua-2 does ML semantic pruning, Caveman handles prose, session-dedup strips repeats across turns. Critically, code, URLs and JSON are preserved byte-perfect, and a default-on inflation guard throws the compressed version away and sends the original if compressing would actually grow the prompt — it never makes things worse. On tool-heavy sessions that's ~89% average input-token reduction (an 8k-token git diff becomes a few hundred). Full credit to every upstream project (RTK, Caveman, LLMLingua-2, Troglodita) is in the README. Agent-native — the agent can drive the router itself. There's a built-in MCP server (95 tools across 30 audited scopes, over stdio / SSE / streamable-HTTP), plus A2A (v0.3, JSON-RPC 2.0) support. That means an agent can query providers, switch combos, read its own remaining quota and manage memory through the gateway — not just consume tokens through it. For context on whether it's worth your time: it's grown to ~9.8K GitHub stars, 1,490+ forks and 280+ contributors in ~4.5 months, with 21,000+ automated tests and 1,830+ issues closed — so it's a battle-tested project, not a brand-new experiment. Happy to go deep on the routing engine, the honest free-tier math, or how the compression pipeline decides what's safe to compress. Repo + install in the first comment. submitted by /u/ZombieGold5145 [link] [comments]

Happy 250th America, here's 5% of OpenAI

OpenAI floated giving the Trump admin a 5% stake. Financial Times ran it citing two people familiar with the talks. OpenAI haven't confirmed or denied anything. $852 billion valuation at last count, March 31. That 5% works out to $42.6 billion in paper equity nobody can touch yet. The sequence is what sticks. Six weeks ago NOTUS had senior officials already talking AI equity stakes with major companies. Three weeks ago Commerce spent 18 days reviewing Anthropic's Fable 5 and Mythos 5 before lifting controls. OpenAI in early formal talks now. I'm old enough to remember when tech got regulated by hearing about it on the evening news months later. Now the regulation happens in parallel, while the product is still being built. The Alaska Permanent Fund comparison keeps surfacing — Americans getting a cut of AI returns the way Alaskans get oil dividends. Shows up in secondary reporting and OpenAI's own earlier policy docs on public wealth sharing. Altman may never have said those words in these talks. We don't know that for sure. There were no governance channels for this six months ago. They're being built out of nowhere — equity stake, export controls, model reviews with fixed timelines. Everyone keeps asking whether Washington gets a seat at the table. Nobody asks what happens when they actually show up and talk money. submitted by /u/roll0ver [link] [comments]

Making Optimization Work When Labels Are Scarce [R]

https://www.gnosyslabs.com/case-studies/safety-classifier-sparse-labels Gnosys is an autonomous model engineer: it improves prompts and classifiers when ground truth is too sparse for conventional optimization. On ToxicChat, a public safety benchmark, under realistic label scarcity, it improved a classifier past both the team's starting point and GEPA (a standard prompt optimizer), across two runs of our current method. This note describes what we did, what we found, and where the method underperformed. Results We report harm caught: the share of harmful messages flagged, holding the false positive rate fixed at 5% (one in twenty) for every method, so a difference reflects additional harm caught at the same cost rather than a change of threshold. Both runs below are scored on a held-out set the system never saw. Headline run (3,000) Prior run (1,000) Gnosys 0.777 0.909 Starting classifier 0.731 0.788 GEPA 0.702 0.848 In both runs, Gnosys improved on both the starting classifier and GEPA. In the headline run GEPA not only trailed Gnosys but fell below the starting classifier (0.731 to 0.702); in the prior run it improved on the starting point. This inconsistency is the central difficulty under sparse labels: optimization sometimes helps and sometimes harms, and without trustworthy measurement there is no way to tell which has happened. The comparison is intentionally conservative: both approaches use the same underlying optimizer. The only difference is that Gnosys engineers the objective the optimizer works against. The problem Teams running high-stakes AI classifiers, in content moderation, fraud, claims review, and risk scoring, share one constraint: the ground truth they need is a human judgment that is expensive, slow, and sometimes never arrives. They can verify only a small set of examples while decisions accumulate on everything else. Tuning the model against the few labels on hand is where the difficulty concentrates. Here "few" is literal: about 200 verified labels, of which roughly 8 were actual harm, against several thousand unlabeled messages. With that little verified signal, an optimizer fits the noise in those examples rather than the underlying pattern, and the direction it moves depends on which handful of labels it happened to receive. How Gnosys is different GEPA improves whatever evaluation signal it is given. That is its job, it does it well, and Gnosys uses it. But Gnosys goes further. As an autonomous model engineer it judges whether the available signal is trustworthy enough to optimize against, engineers a better objective from the sparse labels when it is not, and rewrites the prompts and classifier against that objective. Prompt optimization is one step in the loop. Gnosys automates the entire engineering cycle. Rather than trusting a handful of labels directly, Gnosys fuses the small verified set with the large unlabeled pool into a calibrated estimate of quality, with per-slice calibration and an explicit check that flags when the signal is not trustworthy enough to act on. In both runs, optimizing against that calibrated objective improved on both the starting classifier and GEPA using the same labels. The evidence, slice by slice The figures below are computed against the held-out test labels, full ground truth a deployment would not have. They are point estimates on small positive subsets, so we report the count alongside each, and they are not estimates the system produced from the sparse labels. Because a single aggregate can hide a regression within a category of interest, we report every slice, including losses. All figures compare Gnosys against GEPA on the headline run. By message length (a complete split of the test set): Length Harmful examples vs. GEPA Short (under ~80 characters) 81 −18.5 pts Medium 51 +21.6 pts Long / multi-step (200+ characters) 106 +20.8 pts By harmful-content category (a safety team's working slices): Category Harmful examples vs. GEPA Violence-related 21 +23.8 pts Jailbreak attempts (independently verified) 49 +8.2 pts Sexual content 63 −7.9 pts The gains concentrated where judging the content requires the most reasoning: violent intent, deliberate jailbreaks, and longer multi-step messages, where thin labels leave a standard model guessing. Two slices moved the other way, for different reasons. Short messages, the largest slice, were not a model failure: Gnosys ranks short-form harm at least as well as GEPA. The lower recall is the operating point doing its job. Under a single false positive budget the aggregate-optimal threshold pools alarms where harm is densest, which is longer messages. Setting a budget per segment lifts short-message recall to about 0.90 but lowers the aggregate from 0.78 to 0.71. Sexual content was a genuine limitation: on this small slice (63 harmful of 77 messages) the model ranked worse, and a slice-local threshold would not recover it. These regressions su