Thanks to Amazon Bedrock, We Stopped Losing Money

Read this article on LinkedIn

Most startups building on cloud solutions divide total cloud spend by total revenue to get a blended margin. If that looks healthy, they don’t dig deeper. The problem with that calculation is that it hides how cost is distributed across your customers. Before LLMs usage patterns were easier to model and per-customer cost didn't swing much. With LLMs, that's changed. Inference cost varies enormously from one customer to the next, and you can't price intelligently if you don't know your cost to serve each individual customer.

It's still early days for Trovia, and after we launched our first batch of customers we found one of them was running about 40% of our total inference cost while paying the same rate as everyone else. At scale, we'd be spending roughly four times what we were charging them. Thanks to Amazon Bedrock, we caught this early.

Our engineering team came from AWS so watching cost per customer, not just cost in aggregate, is a habit we brought with us. The setup is simple. Here are the four takeaways for other AI builders:


  • Pick a provider that tracks model usage per customer

  • Tag every workload for the business, not just for engineering

  • Aggregate the data somewhere you actually look

  • Do this early

1. Pick a provider that tracks model usage per customer

One of the main reasons we chose Bedrock over going direct to the model provider is that Bedrock lets you create inference profiles, meaning that you can track model usage per customer. Without Bedrock, per-customer attribution is something we'd have had to build and maintain ourselves, which would’ve been costly and complicated. Instead, using AWS’s cost allocation tags, these costs show up automatically in our AWS Cost and Usage Report.

2. Tag every workload for business, not just engineering

Most teams only track user-facing query traffic. Each of our customer profiles tag the following:


  • Ingestion

  • Validation and evals

  • Product queries

  • Scoring

  • Agent fan-out


The true usage cost per customer isn’t just their traffic, but whatever AI usage was executed by our team on the customer’s behalf or in order to service them.

3. Aggregate the data somewhere you actually look

By using Bedrock, the customer inference profile data is already in our workflow.

The Cost and Usage Report lands in our warehouse and populates our customer table tracking per-customer cost over time, by workload. Here is where this particular customer surfaced. Since it's after-the-fact by a day or two, we pair it with AWS Budgets and anomaly detection that alert us when something unusual happens. By setting up a profile per customer, usage data like latency, errors, throttling, and token throughput, show up in CloudWatch as separate customer dashboards.

The best part is that neither of these require manual input. As long as the inference profiles are tagged with the labels you want to track, the data will show up in the dashboards you’re already using.

4. Do this early

Most teams don't think about this in the early days because cost-per-customer usage does not directly affect the end-user experience. When you are starting out, you are iterating quickly and more customer usage seems like a signal in the right direction.

This thinking becomes unsustainable once you have a few key customers. Without sufficient visibility, pricing and operations can get thorny really fast.

For pricing, once we could see the dispersion, we moved off the flat rate and repriced this customer in line with what they actually cost to serve. Now we can price every account against its real cost, something we couldn’t do before. For operations, if something goes wrong, we’ll already know exactly which customers are affected and how badly.

These recommendations are for any team building an AI product. Set up the following things early: have a way to accurately assess how much each customer costs, clear stats on usage per customer, and an easy way to track them.