prep blog post for AI cost inferencing#542
Merged
Merged
Conversation
✅ Deploy Preview for opencost ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Signed-off-by: Dee Zeis <139172953+dzeiskubecost@users.noreply.github.com>
Signed-off-by: Dee Zeis <139172953+dzeiskubecost@users.noreply.github.com>
Signed-off-by: Dee Zeis <139172953+dzeiskubecost@users.noreply.github.com>
I added an image and made some minor grammar/punctuation edits throughout. One call out - at the end we mention a guide that was developed. Is this live online somewhere we can hyperlink to? Signed-off-by: Dee Zeis <139172953+dzeiskubecost@users.noreply.github.com>
ameijer
commented
Jul 20, 2026
|
|
||
| ## Where things stand | ||
|
|
||
| A proof-of-concept was implemented and tested on a cluster with 109 GPUs and 30 deployed AI models, and the generated metrics were validated. |
Member
Author
There was a problem hiding this comment.
Suggested change
| A proof-of-concept was implemented and tested on a cluster with 109 GPUs and 30 deployed AI models, and the generated metrics were validated. | |
| ** This feature is available in OpenCost 1.121.0!** | |
| A proof-of-concept was implemented and tested on a cluster with 109 GPUs and 30 deployed AI models, and the generated metrics were validated. |
| @@ -0,0 +1,132 @@ | |||
| --- | |||
| slug: opencost-llmd-inference-cost | |||
| title: "Clueless About the Cost of AI Model Usage? OpenCost + llm-d = First-of-a-Kind Kubernetes Inference Cost Tracking" | |||
Member
Author
There was a problem hiding this comment.
Suggested change
| title: "Clueless About the Cost of AI Model Usage? OpenCost + llm-d = First-of-a-Kind Kubernetes Inference Cost Tracking" | |
| title: "OpenCost 1.121.0: First-of-a-Kind Kubernetes Inference Cost Tracking" |
…into atm/blog-post * 'atm/blog-post' of github.com:opencost/opencost-website: Update index.md delete temp Blog image upload Create temp # Conflicts: # blog/2026-07-01-opencost-llmd-inference-cost/index.md
simanadler
reviewed
Jul 21, 2026
| > **Cost ≠ Price:** SaaS providers may price below cost to gain market share or well above cost for premium models. When comparing self-hosting costs to SaaS, keep this distinction in mind. An enterprise's cost for SaaS inference is the provider's price. | ||
|
|
||
| To close this gap, we are integrating [OpenCost](https://opencost.io), a CNCF incubating project, with [llm-d](https://llm-d.ai), a CNCF sandbox project for distributed LLM inference on Kubernetes. This post explains how the integration works, what metrics it produces, and how platform teams can use them to make data-driven decisions. | ||
|
|
There was a problem hiding this comment.
Can we add the following?
Note: vllm users who do not use llm-d can also benefit from these capabilities, since the core metrics used to calculate model and per token costs come from vllm.
simanadler
reviewed
Jul 21, 2026
|
|
||
| The integration uses metrics already present in an llm-d deployment: token throughput from vLLM (`vllm:prompt_tokens_total`, `vllm:generation_tokens_total`), GPU costs from OpenCost's existing allocation engine, and processing-time metrics that support separate cost calculations for input and output tokens. | ||
|
|
||
| The result is a new set of inference cost metrics published to Prometheus and available via OpenCost's REST API and MCP server: |
simanadler
reviewed
Jul 21, 2026
|
|
||
| Kubernetes resource allocation has always had an efficiency problem. Teams size CPU and memory requests based on estimates, which often leads to overprovisioned workloads and silently accumulating idle capacity. With GPUs, the cost of getting this wrong is an order of magnitude higher. Measurement tooling hasn't kept up, making it difficult to measure efficiency or ROI. | ||
|
|
||
| A GPU actively serving a loaded LLM is expensive. A GPU sitting idle with a model loaded but no requests arriving is equally expensive, because the model's weights occupy VRAM whether or not inference is happening. For example, a low-traffic model might spend 95% of its time in this "warm but idle" state, burning through your budget while contributing zero productive tokens. |
There was a problem hiding this comment.
In addition, KV cache hits directly impact the cost of processing input tokens and thus must be measured as well.
peatey
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
a blog post about our ongoing AI costing initiative