.png){kind=logo}
Inference has settled into engineering budgets as a recurring operating expense, sitting beside cloud storage and observability on the invoice. Estimates cited by Brookings place 80–90% of AI computing power on inference, with training accounting for the remainder. Every session, agent loop and retrieval call adds tokens to a meter running continuously.
The International Energy Agency reports that data center electricity demand grew 17% during 2025, with AI-focused facilities surging 50%, and its base case sees consumption roughly doubling to 945 TWh by 2030. The four largest US hyperscalers have guided toward roughly $725 billion in 2026 capital expenditure, up about 77% year over year.
Engineering teams have responded by treating inference as a procurement problem with several independent levers, covering model breadth, purpose-built silicon, decentralized compute, serving-engine efficiency, and unit price. The five entries below represent those levers, ordered by adoption instead of cost.
Inference accounts for the majority of AI compute, making per-token pricing an operating expense that compounds with adoption.
Open-weight model APIs generally run 50–90% below frontier pricing for comparable workloads.
Speed-optimized silicon and cost-optimized GPU fleets solve different problems, and the cheaper option carries a latency penalty.
Decentralized networks publish the lowest headline rates, with model coverage and price stability needing validation.
OpenAI-compatible endpoints have cut switching costs to a base-URL change, making multi-provider routing realistic.
Together AI serves the broadest open-weight catalog among major serverless platforms, with more than 200 models spanning chat, vision, embeddings, image, and audio behind one endpoint. Published rates span roughly $0.03 to $9 per million tokens, with smaller Llama, Qwen, and Mistral variants at the low end.
Fine-tuning differentiates the platform for teams needing custom behavior without dedicated infrastructure, since adapter-based inference runs at standard rates plus a modest overhead. Dedicated endpoints and GPU clusters cover steady-state traffic.
Type
Serverless inference and fine-tuning across a broad open-weight catalog
Best for
Model breadth, custom fine-tunes, and competitive pricing at volume
Pricing
Per-token, varying by model, with reserved capacity lowering effective rates
Drawback
Per-token cost sits well above decentralized networks, and enterprise agreements carry a premium.
Groq runs open-weight models on custom Language Processing Unit (LPU) silicon built for sequential token generation. Independent measurements place throughput between roughly 300 and 1,000 tokens per second, against 50–150 for comparable GPU serving, and Llama 3.3 70B is published at $0.59 input and $0.79 output per million tokens.
Speed of that order changes what is buildable, benefiting real-time voice agents, coding assistants, and conversational interfaces. A batch API at 50% off and prompt caching lower effective rates for asynchronous work.
Custom LPU inference hardware
Best for
Lowest-latency real-time chat, voice, and coding assistants
Pricing
Per-token from around $0.05 per million input tokens, competitive on speed-adjusted cost
Drawback
A deliberately curated catalog optimized for speed, with no support for customer fine-tuned deployments.
Gonka pools independently operated GPUs behind independent brokers, each exposing an OpenAI-compatible endpoint. Eleven are listed in the developer docs, and any operator can run one. A Transformer-based proof-of-work protocol routes nearly all GPU cycles into productive inference, and a CertiK audit is complete.
Network pricing sits around $0.0004 per million tokens, and has held there since launch, a deliberately minimal test-stage rate. Utilisation-based dynamic pricing remains on the roadmap, so changes arrive through on-chain miner votes, and brokers add their own margin. Two models are served, Kimi K2.6 and MiniMax M2.7.
Type
Decentralized AI compute network for inference, with training planned
Best for
Teams seeking the lowest published cost without vendor lock-in, plus verifiable compute
Pricing
Fixed rate of $0.0004 per million tokens, plus the broker's margin
Drawback
A newer network with a small catalog and broker-mediated access, so coverage, latency, and price stability need validating per workload.
Fireworks AI targets production reliability through proprietary FireAttention kernels and a FireOptimizer auto-tuning layer, applying speculative decoding and workload-aware batching to extract more throughput per GPU than open-source frameworks deliver. Cursor, Notion, DoorDash, and Quora are among its paying customers.
Agentic applications are the natural fit, with function calling, guaranteed JSON output, and grammar-constrained decoding as first-class features. Prompt caching and batch inference each cut token costs by half and compound independently, suiting retrieval-augmented generation (RAG) pipelines with heavy prefix reuse.
Type
Production inference platform built on the FireAttention engine
Best for
Agentic applications needing function calling, structured output, and low latency
Pricing
Per-token, typically $0.20 to $3 per million tokens, with stacking cache and batch discounts
Drawback
Positioned around enterprise reliability, so competing hosts undercut it on shared models.
DeepInfra has pursued one objective since 2022: minimizing the unit cost of open-weight inference. It hosts roughly 77 models on its own US-based infrastructure, including Blackwell B200 systems, at rates spanning about $0.02 to $1.50 per million tokens. Batch inference applies a further 50% discount for jobs tolerating 24-hour turnaround.
Compliance coverage is stronger than the pricing suggests, with SOC 2, ISO 27001, GDPR, and HIPAA compliance in place. Bulk summarization, dataset labeling and evaluation pipelines are where the economics land best.
Type
Cost-efficient serverless inference across a wide open-model catalog
Best for
Bulk and background workloads where unit cost outweighs peak speed
Pricing
Low and predictable per-token rates, with automatic volume tiering
Drawback
Centralized and tuned for predictable cost, so real-time throughput trails purpose-built inference silicon.
Training costs arrive as discrete capital events, whereas inference accrues with every request served. An application handling ten million monthly requests generates a bill scaling linearly with adoption, inverting the software economics where marginal cost nears zero.
Model architecture influences the arithmetic as much as provider choice. Mixture-of-experts designs activate only a fraction of their parameters per forward pass, keeping models such as DeepSeek V3 and Kimi K2 cheap to serve. Output tokens also cost two to five times more than input tokens, so generation-heavy workloads behave differently from retrieval-heavy ones.
Latency requirements should be established before price comparison begins, since interactive interfaces need sub-second time-to-first-token, whereas overnight batch jobs accept slower serving for lower rates.
Pricing structure warrants as much attention as headline rates, because dynamic pricing, subsidized rates, cache discounts, and reserved capacity all change the effective figure. Querying a provider's model-list endpoint confirms genuine availability, and an evaluation set remains the only reliable way to verify quality at a lower price. OpenAI-compatible endpoints make multi-provider routing a sensible hedge against price movement and capacity limits.
This comparison draws on publicly available pricing pages, technical documentation, independent pricing trackers, and third-party benchmark data current as of July 2026. Providers were selected to represent distinct cost-reduction levers instead of a ranked hierarchy, and ordering reflects adoption. Pricing moves often enough that rates should be confirmed directly before any procurement decision.
No entry wins across every dimension, which reflects the structure of the market more than any deficiency among the providers. Together AI offers the widest catalog and clearest fine-tuning path, Groq delivers throughput GPU serving cannot match, Gonka publishes the lowest rates on a verifiable decentralized layer, Fireworks AI serves agentic production workloads, and DeepInfra supplies dependable low-cost bulk capacity.
Teams treating inference as an operating-cost line increasingly route across several providers, matching each workload to the lever that fits it. The right answer depends on workload shape and deserves revisiting as pricing moves.
Why has AI inference become a bigger cost concern than model training?
Training is a one-off capital event for most organizations, since the majority of teams consume models built by others. Inference recurs with every request served, and estimates place it at 80–90% of total AI compute, making it the line that grows with adoption.
How much cheaper are open-weight model APIs than frontier model APIs?
Open-weight models served through Together AI, Groq, Fireworks AI, and DeepInfra typically run 50–90% below frontier API pricing. The trade-off appears on hard reasoning, complex agentic tool use, and frontier-grade code generation, where proprietary models retain a measurable lead.
What should be tested before switching inference providers?
Run an evaluation set covering quality, latency, and failure behavior under realistic concurrency, and confirm the model identifiers available on the endpoint. Since most providers expose OpenAI-compatible APIs, a parallel test costs little beyond engineering time and reveals reliability differences that published figures miss.
