Inference is the use of a trained model to produce outputs from new inputs. Training teaches the model; inference is what happens every time a user sends a request and gets a prediction, classification, or generated text back.
Why it matters
For deployed products, inference is the ongoing cost and the part users feel. It determines response latency, throughput, and hardware needs, and it is where reliability matters: a model endpoint that times out is an outage like any other.
Example
An API call to a hosted model is an inference request:
POST /v1/generate
{"prompt": "Summarize this ticket...", "max_tokens": 100}Time to the first token and tokens per second are the usual speed measures, which you can track much like any other endpoint.
Common mistakes
- Planning only for training cost and underestimating serving cost at scale.
- Not setting timeouts and retries for model endpoints.
- Skipping monitoring of latency and error rate on the inference service.
Related
Get notified when this term updates
Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.
This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.