Hey team,
I've been working on implementing a Retrieval-Augmented Generation (RAG) pipeline, and I've finally gotten a good grasp of the cost implications for each component — but I'd love to hear if my experience aligns with yours.
Here's the breakdown as I see it:
Embeddings: We're using OpenAI's embeddings API. At 0.0004 USD per 1k tokens, we've found that this cost quickly adds up when processing large documents. For example, processing around 10GB of text files ended up costing about $80 monthly just for initial embedding creation.
Vector DB (Pinecone): Storage and query costs are our biggest factors here. Storing the embeddings costs roughly $0.06 per vector monthly. For a mid-sized dataset, this is about $150. Query costs vary, but with a typical load, we're looking at another $200/month.
Inference: Using GPT-3.5 turbo for generation at $0.002 per 1k tokens gets expensive fast, especially considering the upward of 100k tokens per month during peak usage. The monthly cost stacks to around $600.
I've seen suggestions to run open-source LLMs or embedding models on-premises to cut down costs, but does anyone have experience with that? Particularly curious about running something like sentence-transformers locally, and if the investment in hardware would actually bring savings.
Looking forward to hearing your setups and suggestions!
Have you considered reducing token usage by preprocessing your text data to remove unnecessary information prior to embedding? This can help cut down on your OpenAI API costs significantly, especially when dealing with large corpora.
I've been experimenting with running sentence-transformers locally, and for us, it did cut down on costs significantly. We set it up on a dedicated server with a couple of 3090 GPUs. Initial setup and hardware cost were a bit steep, but it pays off in the long run, especially since we're inclined to run many iterations. The flexibility and control over the process have been a huge plus!
I've had a very similar experience with embedding costs! We also use OpenAI's API, and when dealing with large datasets, the costs can spiral quickly. One tip I can share is pre-processing your data to eliminate any unnecessary text, which helped us cut down our costs a bit. As for running models on-premise, we've been considering it too, but the initial hardware investment is holding us back.
That's an interesting breakdown! Have you considered using alternatives like the FAISS library instead of Pinecone for the vector database? It might offer better cost-efficiency if you're okay with an open-source solution. We've been using FAISS for a few months, and while it requires some initial setup, the operational costs are quite minimal, about $50/month for the same data scale you're handling.
Totally agree with your breakdown, especially on the OpenAI embeddings. I had a similar experience, and our costs escalated quickly. We switched to using Sentence Transformers locally, which reduced the embedding cost significantly. We invested in a decent GPU setup, around $2500 one-time expense, and it’s been worth it. You just need to ensure you have the right infrastructure to support it.
Have you considered using other vector databases like FAISS or Milvus? FAISS is open-source, and although it requires more setting up, it can be cheaper in the long run if you go the on-prem route. We use Milvus with Kubernetes, and it has been pretty efficient, especially for larger datasets. Worth a look if Pinecone gets too expensive!
For our mid-sized project, we've been using FAISS as an alternative to Pinecone. The upfront effort was certainly more involved, but the long-term storage and query cost savings are substantial. Our monthly costs are under $100, including both storage and queries, after initial setup and optimization.
Hey, I totally agree with your breakdown. We've been using OpenAI's embeddings as well, and those costs do stack up quickly with large datasets. We've experimented with local setup using sentence-transformers on a decent GPU, and it does bring down costs significantly in the long run, but the initial hardware investment was hefty (around $3k) for our use case.
Have you tried Faiss for vector similarity searches? It's open-source and can be a more affordable alternative to Pinecone if you're willing to manage some infrastructure on your end. You might need a decent on-prem setup, but I've found the cost savings to be worth the initial effort.
Curious about your inference process—are you batching requests or doing them one at a time? I've found that batching can really help manage token usage and reduce costs.
I've been running a RAG pipeline with Hugging Face's sentence-transformers locally. While the upfront cost for GPU hardware is steep—our setup was around $2,000—it significantly cut our recurring costs. We managed to save around $100/month on embeddings alone compared to previous API costs. Running locally also gives us more control over the scale and performance, but it does require some additional maintenance and expertise to manage the infrastructure.
