Hey folks,
I'm in the process of optimizing our LLM spend and noticed that tracking costs across different providers (OpenAI GPT-4, Anthropic Claude, Cohere) isn't as straightforward as I'd hoped. We're currently juggling multiple APIs for various projects, and keeping tabs on each one is becoming a nightmare.
I've been testing some observability tools like Elastic APM and Datadog, but I feel they cater more towards broader application monitoring rather than focusing on specific LLM consumption. Grafana Loki integrated with Prometheus has been useful for basic logging and metric collection, but I'm curious if anyone has found a more specialized tool that offers better insights into LLM usage patterns and helps track spending more efficiently across multiple providers.
Also, has anyone successfully built a custom solution using AWS CloudWatch or GCP's Monitoring for this specific purpose? Would love to hear your strategies and recommendations!
Thanks!
Cheers, Alex
If you're open to trying new tools, consider experimenting with Kubecost if you're deploying on Kubernetes. While it's primarily used for Kubernetes cost management, with some customization, it can help track resources and costs for your LLM instances.
I'm curious how often you're checking your usage and cost reports? We've automated daily reports with Google Sheets and their API integrations as a stopgap solution. Not ideal, but at least it gives us a heads-up on daily trends.
Hey, have you tried using New Relic? We shifted to it recently for our LLM usage and it's been pretty effective. It has better granular reporting on API call costs, especially when you set up custom metrics and alerts tailored for LLM usage. It’s a bit of work initially, but the clarity it offers has made our cost optimization efforts much easier. I’ve seen a 15% reduction in monthly overhead since using it, due to the insights we now have on usage patterns.
We went a similar route with Google Cloud Monitoring and Cloud Logging. The primary challenge was normalizing data across providers, but with a few custom scripts, it’s manageable. One thing that worked well for us was setting up alerting rules for sudden usage increases. However, I’m also curious if there’s a more out-of-the-box solution tailored for LLMs these days.
Hey Alex, I've been in the same boat. We ended up building a custom dashboard using AWS CloudWatch with Lambda functions to parse and aggregate cost info from the API usage logs. It's fairly low-level, but it gives us the flexibility to pull specific metrics we need. It took some tinkering, but it's been worth it for the granular control.
Hi Alex, I'm in the same boat and completely feel your pain regarding the complexity of tracking LLM usage costs. I have built a custom solution using AWS CloudWatch where we parse the API costs from logs and consolidate it into dashboards. It took a bit to set up initially, but it has been working well for us. However, it still lacks some real-time insights that I wish it had.
Hey Alex, I totally feel your pain. We experimented with combining Grafana with AWS CloudWatch for more granular monitoring. By setting up custom metrics in CloudWatch to track API usage and costs specifically for each LLM provider, we were able to create dashboards that gave us a better real-time view. It took some initial setup, but it's been helpful in curbing unexpected spending spikes.
