Hey folks,
I've been diving into LLM observability tools recently, trying to figure out how we can better track our spending across different providers like OpenAI, Cohere, and Anthropic. We've been using GPT-4 fairly extensively, and costs are starting to creep up. Not to mention, we're considering trying out some models from other providers, but I want to have solid observability in place first.
Does anyone have experience with any of these tools? I've looked into Weights & Biases for general model tracking but wasn't sure if it covers spend effectively. I've heard about new tools like MosaicML's platform that promise cost efficiency and precise usage stats – but not sure if they integrate well across different LLM providers.
What tools are you all using to track and manage LLM usage costs effectively across multiple providers? Any insights or recommendations would be hugely appreciated, especially with any integration tips for merging these tools into existing observability setups!
Cheers!
Have you looked into LangChain for integrating model providers? While it's more focused on chaining, their community is talking a lot about observability lately. For tracking costs, I rely on Stripe's API internally alongside Weights & Biases for metrics. It's not a single solution but combining them has provided all the insights I need. Wondering if anyone else has managed to centralize this more efficiently?
I've been in a similar boat as you with trying to better track LLM spend. We've actually started using TruEra which offers a comprehensive dashboard for LLM observability. It integrates pretty well with OpenAI, Cohere, and Anthropic, and has been great for cost tracking. You get breakdowns of usage and costs by provider, which is a nice plus. One tip: take the time to set up custom alerts for when cost thresholds are exceeded. It's saved our budget more than once!
Has anyone tried using Prometheus with custom metrics for tracking these? We're thinking of extending its functionality to capture specific spend metrics, but not sure how granular it can get with providers like Anthropic or Cohere. Any experiences or insights?
We’ve been using MosaicML lately, and it's been extremely helpful for tracking costs across various LLM providers, not just OpenAI. The platform provides detailed breakdowns and integrates smoothly with our existing infrastructure through APIs. I'd recommend giving it a shot, especially if you're considering other providers.
I'm curious about how you've been managing cost spikes with GPT-4? Have you tried any throttling or usage quotas to keep costs predictable? Also, would love to hear if anyone's using Google Cloud's new AI observability toolkit. Heard good things about it giving detailed spend breakdowns but haven't tried it myself.
For us, it was critical to have a platform that monitors costs in real-time. After a bit of trial and error, we found that using a combination of Prometheus for general observability and a custom script polling APIs for billing gave us a pretty solid overview. This setup requires some maintenance but gives us the flexibility to handle data from different providers.
We've been using Weights & Biases and while it's great for tracking experiments, it doesn't quite give us the granularity in billing data across multiple LLM providers. Recently tried out Langchain as a framework because its integrations seem to handle multiple LLMs effectively, though it's not strictly for cost tracking.
