Hey folks, I'm currently working on deploying a custom Large Language Model for my company, and I need some insights on how to manage it efficiently, particularly in terms of cost. We're using OpenAI's GPT-3.5 due to its balance between performance and price. However, I'm concerned about the expenses as the usage scales up. Here are some details:
Deployment Architecture: We're running everything on AWS, making use of EC2 instances for compute resources. We initially chose on-demand instances, but costs are spiraling as we scale up.
Usage Patterns: The model is primarily used for generating customer support responses and internal document summarization. A lot of the usage occurs during peak business hours, which adds to the cost.
Optimizations Tried: We experimented with spot instances and a combination of CloudWatch and Lambda to automate scaling based on traffic. This helped a bit, but the unpredictability of spot instance availability poses another challenge.
I'm keen to hear if any of you have tackled similar challenges or if there are additional tools and strategies (perhaps Kubernetes for automated scaling or cost monitoring tools for better allocation) you've found useful for optimizing costs? Also, are there alternatives to spot instances that provide both reliability and cost-effectiveness?
Thanks in advance for any advice you might have!
