Hey folks,
I've been running multiple LLMs from OpenAI, Cohere, and Anthropic to compare performance and utility for various tasks we're handling. It's been eye-opening, but the hardest part for me so far is tracking and managing the spend across these platforms.
I was wondering if anyone here has experience with observability tools that are particularly good at keeping tabs on LLM-related costs? I tried using Prometheus for a custom setup, but it got a bit too complex and still didn't quite give me the granularity I needed. I've read about commercial tools like Datadog and New Relic, but I'm not sure if they cover all my bases when it comes to specific LLM usage and cost allocation.
Any suggestions or experiences to share? Would love to hear how others are handling cost tracking and observability in a multi-provider LLM environment!
I haven't used Datadog for LLM cost tracking specifically, but we've had success with it for other cloud cost management purposes. It might be worth reaching out to their support to see if they have any specific integrations or plugins for the providers you're using. Alternatively, AWS Cost Explorer offers some interesting insights, though it's more suited if you're primarily on AWS.
I've used both Datadog and New Relic for this purpose, and while they're great for general observability, they're not as fine-tuned for tracking specific LLM-related costs. You might want to check out Cortex or Grafana with Loki for a more customizable solution. They require more setup, but can be tailored to track specific metrics across different providers.
I've been in a similar boat with OpenAI and Cohere. What worked for us is using Grafana alongside Prometheus for better visual insights. We set up custom dashboards that break down costs per API call and provider. It needed some initial manual setup but eventually gave us the granularity we need.
I've been in a similar situation with multiple LLM providers, and I found that using OpenTelemetry has been pretty helpful. It required some initial setup work, but it provides enough flexibility to track specific usage metrics across different providers. You might still need to pair it with a dashboarding tool like Grafana for better visualization, but it could give you the granularity you're looking for.
Interesting that you mentioned New Relic. I've actually used it for cost tracking in a multi-cloud setup and while it wasn't initially geared towards LLMs, their custom metrics and logging features could be adapted. However, it's not out-of-the-box, so some tweaking is needed.
Has anyone tried out CloudWatch for this? We're a heavy AWS shop and are considering sticking to it for its seamless integration with other AWS services. Curious if it can serve the kind of granularity you're looking for since it’s less talked about in the context of LLMs.
I can relate to the challenges you're facing. In my team, we created a custom dashboard using Grafana and integrated it with individual billing data exports from each provider. It involves some manual effort initially but gives us a lot of control over what we monitor. We saw a 15% reduction in unexpected costs once we got everything up and running. If you're comfortable with some scripting, it's worth exploring.
I faced a similar issue when dealing with multiple providers. I ended up using Grafana paired with some custom scripts to pull in detailed billing reports and usage metrics. It requires a bit of setup, but I now have dashboards showing real-time costs. It also helps to set alerts for when usage crosses certain thresholds. Definitely worth the initial setup time!
I've been using Grafana coupled with Prometheus for monitoring, but I totally get what you mean about the complexity. We actually switched to CloudWatch when dealing with AWS-hosted solutions, and while it's not perfect, it does a decent job of centralizing our metrics. For cost-specific tracking, I recommend checking out some of the newer tools like Kubecost—it’s pretty good for Kubernetes-related metrics and could give you some visibility into individual service costs if you're deploying through k8s.
We're running a hybrid setup and have found that custom dashboards in Grafana, along with real-time alerts based on anomaly detection, have helped to keep costs in check. It's not bulletproof, but we can tweak things as necessary. Out of curiosity, has anyone tried using something like Anodot for anomaly detection specifically in the LLM usage and cost sphere? I’m curious if anyone has some benchmarking comparisons they can share.
Have you tried Splunk? I've used it to track costs across different cloud services before, and while it's a bit pricey, the customization options are vast. It should be versatile enough for multi-provider environments. Also curious about how much variance you see in costs between the providers?
