3 cost saving tips for businesses implementing generative AI

In this article, I will outline those pitfalls and how you can achieve AI transformation at an optimal cost for your business.

For large language models (LLMs), interactions are often served via Application Programming Interfaces (APIs) provided by several vendors such as OpenAI, Anthropic, Amazon Bedrock, Groq, and others.

For example, as of now, OpenAI’s GPT-4 model API calls cost $2.5/mTok for input tokens and $10/mTok for output tokens. Output tokens generally cost more than input tokens because LLMs utilize significantly more computing resources when generating responses.

1. Caching

If you are building your AI solution internally, leveraging the LLM’s built-in prompt or context caching is highly effective. OpenAI, for instance, automatically implements caching for input tokens exceeding 1024 tokens. Other providers like Anthropic and Google also have caching mechanisms detailed in their API documentation.

Caching means that if 10 users ask your AI assistant or chatbot the same question, the response from the first request can be reused for the other 9 users. This approach eliminates the need for 9 additional API calls, thereby significantly reducing costs.

Additionally, you can enhance caching with a distributed framework like Redis. Alternatively, using an AI gateway such as Cloudflare’s AI Gateway at the edge of your deployment provides built-in caching and rate-limiting functionalities.

2. Better Prompting

Optimizing your base prompts using prompt fine-tuning tools can substantially reduce input token length.

Furthermore, depending on your use case, always include rules in your input prompts instructing the LLM to summarize its output in the most concise and understandable manner.

3. Do Not Reinvent the Wheel

If a solution already exists and is affordable, there is no need to rebuild it. Small and medium-sized businesses aiming to adopt Generative AI quickly, especially without strong engineering teams, should explore tools like Amazon Q (AWS), Google’s NotebookLM, or Microsoft’s Co-pilot. These tools lower the barrier to entry and deliver quick wins.

For businesses with highly sensitive data requirements, consider implementing on-premise AI solutions like AnythingLLM with Ollama. This allows interaction with a Small Language Model (SLM) running on a CPU within your office, without requiring internet access. Although SLMs offer a lower level of intelligence compared to LLMs, they are often sufficient for most business use cases and come at a much lower infrastructure cost.

Optimizing costs for your AI transformation project is critical to its success. Cost-effective solutions not only make implementation more feasible but also help win over stakeholders in organizations where resistance may hinder adoption.

Explore the available tools to accelerate adoption and always review the data usage policies of the solutions you choose.

Explore the available tools to accelerate adoption and always review the data usage policies of the solutions you choose.