Helicone

Helicone AI

Helicone AI

Helicone AI

Helicone AI

Helicone AI

Helicone AI

Helicone AI

Helicone AI

Helicone AI

Helicone AI

Building independent LLM drift detection - sharing the methodology, looking for feedback on the approach

Disclosed upfront: I run [Tickerr dot ai], an independent external monitor for AI APIs. Today it tracks latency, TTFT, uptime, and error rates across major models. I’m trying to validate a more specific idea before building too much. Basic transport health is not the hard part. If Claude/OpenAI/Gemini gets slow, times out, or throws 5xx errors, most teams can catch that with APM, logs, Sentry, Langfuse, Helicone, Datadog, etc. The harder failure mode seems to be silent model behavior drift when API returns 200, latency is normal, no exception is thrown, output looks plausible, but JSON adherence, tool-calling, refusal behavior, reasoning quality, or instruction-following has quietly degraded. This gets worse with agentic systems. In a normal chat, drift may produce a bad answer but in an agentic workflow, the model can silently choose the wrong tool, stop early, mark a task as complete, or take a bad action while everything still looks successful at the API level. The system is running and confidently doing worse work. User complaints are still the primary detection mechanism currently for these. VIGIL (arXiv 2605.08747) found 65 to 88 percent of false-success reports happened at literally zero task progress. DeployBench (2606.05238) found most failures were the system stopping against a softer bar it set for itself and returning clean. Plausible-in-isolation is the failure mode itself, not a sign you are safe, which is why a single model's output never alerts on its own. That's what I'm thinking to build - an external drift detection probe on top LLM APIs, that stays out of your system and does continuous checks every hour, to find out these silent degradations, and sends proactive alerts. Rough idea: External canary suite: run private fixed prompts on a schedule against major models. Track schema adherence, instruction-following, refusal/over-refusal, output length, tool-call format, and simple deterministic correctness checks. Drift baseline: Do not judge a single output in isolation. Track whether today’s behavior has materially shifted versus that model’s own baseline. Cross-model comparison: For some task types, compare model behavior against peer models. Not to say which model is “right”, but to detect abnormal divergence. Example: “Sonnet and Gemini usually disagree 12% of the time on this task type; today disagreement is 28%.” Optional bring your own prompts: A paid tier where you provide some critical prompts from your own workload. Tickerr runs them on a schedule and alerts if behavior drifts from your baseline. Prompts would remain private and would not be public benchmark prompts. What I’m trying to learn: Is this technically sound enough to be useful, or are there are other failure modes that I am missing / are more valuable ? Which alerts would you actually care about? JSON/schema adherence drift tool-call format drift refusal/over-refusal drift output length drift cross-model disagreement spike bring-your-own-prompt regression alerts Would you pay for this, or would you just build it yourself? If you would pay, what pricing feels realistic? $19/month $99/month $299+/month for team/Slack/webhook/BYO prompts Brutal feedback welcome. If this is not a real pain, I’d rather know now, or which direction you feel makes more sense to take this. submitted by /u/Remarkable_Divide755 [link] [comments]

I built a local CLI to estimate and cap AI coding-agent spend before a run gets expensive

I build apps with coding agents, and one thing kept bothering me: before starting a run, I often had no idea what it might cost. Sometimes the agent is useful. Sometimes it keeps retrying the same bad path, rewrites its plan, burns tokens, and only later I realize that the run was more expensive than expected. So I built Runcap. It is a free MIT local CLI for developers using AI coding agents. The idea is simple: estimate a run before starting set a hard budget cap run a local gateway that can stop over-budget calls compress logs / JSON / stack traces before forwarding record what happened during the run generate a rescue prompt when the agent gets stuck It is not trying to replace Langfuse, LiteLLM, Helicone, or other observability/gateway tools. Those are useful, but I wanted something smaller and more direct for my own workflow: a local “cost seatbelt” before a coding-agent run gets out of control. Install: npm install -g runcap GitHub: https://github.com/kirder24-code/ai-agent-manager It is still early and probably rough. I would really appreciate feedback from people using Claude Code, Cursor, Codex, Aider, or other coding-agent workflows. Main question: would you actually keep a tool like this running day to day, or is this too much friction for your workflow? submitted by /u/Ok-Serve4908 [link] [comments]

[P]Stop using print() to debug your agents. Here's a 60-second alternative.[P]

Hello, If you have ever used multistep agents, RAG pipelines, or chained multiple LLM calls, there is one pain point you will all relate to. When an agent gets stuck in an infinite loop, suddenly hallucinates on the third step, or is quietly burning through OpenAI API credits... tracing exactly where the problem originated is a real nightmare. Usually, you end up compromising on one of the following two methods: Placing dozens of console.log or print() statements all over your once-clean code. Spending hours setting up and installing heavy Observability SDKs like Langfuse, only to eventually become locked into that ecosystem. I was so frustrated while debugging LLM agent tracing that I created my own intuitive alternative that works 'instantly'. The key is simply replacing the baseURL. 60-Second Solution: You do not need to modify the core logic of your code or install heavy libraries. Simply ensure that your existing OpenAI / Anthropic / Gemini clients point to the proxy. https://preview.redd.it/dlgok064fa5h1.png?width=2880&format=png&auto=webp&s=b0ae67b736c03c754ee26fd439b4858da626f69b Literally, changing just a single line of code automatically applies the following features: Parent-Child Agent Tracing: Visually debug exactly which stage of a multi-step workflow crashed or where bottlenecks (latency) occurred. Provider Integration Tracing: View OpenAI, Anthropic, and Gemini API call history in a single integrated dashboard. Perfect for teams using multiple LLMs. Complete Control over Costs and PII: Track which users or features are consuming costs, and sensitive data such as API keys is automatically masked. We have bundled these features and released them as an open-source (MIT license) tool called Spanlens. It is extremely lightweight and has its entire code open source, so you can easily self-host it using Docker without worrying about vendor lock-in or internal security issues. If you are tired of messy log debugging and the unpredictable LLM API charges that arrive at the end of every month, please check out the GitHub repository. https://github.com/spanlens/Spanlens I would be very grateful if you could feel free to give me feedback on what tools you are currently using to track complex LLM workflows, and if you have any suggestions for Spanlens after trying it out! This isn't a commercial promotion, just try it for free. I want feedback. submitted by /u/Limp_Shine8489 [link] [comments]

Made an awesome-list for everything LLM cost, would love contributions

So a few months back I got surprised by my Anthropic bill which somehow racked up like $400 ish on a staging key in a few weeks just running evals, no budget cap pretty dumb in hindsight I mean it’s not a big cost but I should have been careful nonetheless After that I started keeping a notes file of tools that actually helped reduce cost stuff like token counters, pricing pages that update properly, caching layers, prompt compression libs, observability tools (helicone, langfuse, langsmith, etc) it slowly grew to 80–90 entries so I cleaned it up and put it on github: https://github.com/ankitvirdi4/awesome-llm-cost what’s in there right now: pricing calculators + token counters observability / tracing (helicone, langfuse, langsmith, openllmetry, phoenix) caching (gptcache, semantic caching approaches) model routers (openrouter, notdiamond, portkey) prompt compression + context window stuff eval cost tracking self hosting / GPU cost calculators everything is linted (awesome-lint), short descriptions for each entry, and I checked links recently so nothing should be dead if there’s anything you’ve used that saved you money on inference, drop it here or send a PR especially looking for more prompt compression stuff, that section feels kinda weak rn not affiliated with anything listed btw just got tired of having 80 bookmarks submitted by /u/OldComposerbruh [link] [comments]

After 6 months of running AI agents in production I think the framework you pick barely matters. The thing that kills them is something else.

Going to get downvoted for this but here we go. I've been running about 30 agents in production for paying customers for the last 6 months and I'm convinced the framework debate is mostly a distraction. LangChain, CrewAI, AutoGen, OpenAI Agents SDK. Pick whichever one your team already knows. It doesn't matter as much as you think. What actually decides whether your agent works in production is something almost nobody talks about on this sub, and it isn't in the framework. Here's what I've seen kill more agents than every framework bug combined. The agent gets stuck in a loop. It calls the same tool 200 times in 4 minutes because something downstream returned ambiguous data and the LLM decided to retry forever. Your OpenAI bill goes from $3 a day to $400 in one afternoon. By the time you notice you've burned a grand. You can't even tell which agent did it because there's no audit trail. Your VPS reboots overnight for kernel patches. Every agent that was mid-task loses everything. Tomorrow morning the support agent has no memory of yesterday's tickets, the research crew has forgotten what they were investigating, the pipeline agent restarts from scratch. None of these are framework problems. They're memory and state problems. A customer complains the agent gave them wrong info three days ago. You go to debug. There's no record of what the agent saw, what it decided, or which tool calls it made. The framework didn't log that because frameworks aren't observability tools. You shrug and refund. You scaled to 15 agents working together. Two of them have conflicting beliefs about the same customer because their memory isn't shared. The customer gets two different answers in the same conversation depending on which agent replies first. You've been around enough times to realize the part you actually need isn't in the framework at all. What I think the real stack is. The framework just orchestrates LLM calls. Use whatever your team likes. It's the cheap layer. A persistent memory layer that survives crashes, restarts, and redeploys, so the agent has actual continuity. This is the layer that decides whether your agent is a toy or a product. Loop detection at the runtime layer, not bolted on as a wrapper around the framework. Something that catches your agent making the same call too many times in a row and stops it before the bill explodes. An audit trail of every decision the agent made, with a hash chain so you can prove later what happened when the customer pushes back. Screenshots and logs aren't enough when ten thousand dollars is on the line. Shared memory between agents in the same team so they're not having different conversations about the same customer. Cost tracking per agent so you actually know which one ran away with your budget. When I look at what makes the agents that survive production look different from the ones that died, it's never that they picked the right framework. It's that they had this layer underneath, either built carefully in-house or borrowed from somewhere. Full disclosure I'm building one of these tools. There are others. Mem0 and Zep and Letta in the memory space. Helicone and LangSmith in the observability space. Mix and match. Use one or build your own. Just please stop arguing about whether LangChain or CrewAI is better when the thing eating your production agents has nothing to do with either of them. What's been your worst production agent failure? Curious what other people have actually hit. I built a free tool that aims to solve most of this issue, what do you think? submitted by /u/DetectiveMindless652 [link] [comments]

How are people actually tracking OpenAI costs in production?

Curious what this community actually uses for OpenAI cost monitoring on real production apps. There are a lot of "I got a $X surprise bill" posts here, but I rarely see the follow-up: what tooling did people land on after the wake-up call? For those running OpenAI in production: - Real-time tracking or just checking the billing dashboard monthly? - Rolling your own or using a tool (Helicone, Langfuse, etc.)? - Breaking costs down per user / per feature, or just looking at the total? Asking because I'm building in this space and trying to figure out what people actually do vs. what they say they should do. submitted by /u/VariousHour7390 [link] [comments]

Agentic Workflow Visualization and API Gateway

I am building an API gateway for agents that can make your agentic AI code model and provider agnostic. I am also grouping agent runs that show multiple llm calls and tool calls in the visualization piece. It gives details on tokens, cost and model latency. I am doing this without requiring any instrumentation in the agentic code. The agents (python for now) are started by a rust correlator that assigns a job_id to each agent so we could track api and tool (inferred from http requests and responses) calls across the entire agentic run. The servers are also in rust. I also have an implementation where instead of the rust correlator i have python and other platform shims that do the same job and the servers are in go. I would appreciate comments from people who are in AI ops who use tools like litellm and Helicone and can provide feedback or complicated use cases. I plan to make everything open source so looking for collaborators too. submitted by /u/High-Speed-Diesel [link] [comments]

OpenTelemetry just standardized LLM tracing. Here's what it actually looks like in code.

Every LLM tool invents its own tracing format. Langfuse has one. Helicone has one. Arize has one. If...