High-performance inference for open frontier models.
Infer rents GPU capacity and runs it hard. Our first listed model is GLM-5.3-Flash, served on OpenRouter against a published speed floor rather than a best-effort average. The gateway — admission control, session affinity, failover — is ours; the hardware is replaceable.
Per-stream decode throughput. This is an engineering policy set in Phase 0, not a measured statistic — no traffic has been served yet. Measured numbers, with their window and timestamp, replace this figure at launch.
The speed floor is a contract, not an average.
Most providers publish a mean and let the tail go where load takes it. Infer sets a floor first and sizes admission around it. Under load, capacity is refused before throughput is allowed to fall through the band.
How the policy works. Admission control counts decode occupancy and uncached prefill rate per replica and across the fleet. When either dimension would push an admitted stream toward the band edge, the next request is refused immediately. The consequence is deliberate: a stream that starts on Infer runs at the advertised rate to completion, and a stream we cannot run at that rate is never started.
Capacity is shed with 429. Never with 5xx.
The three terms below are how we name failure behaviour in this document, not a dashboard. No traffic has been served yet, so none of these states is a current claim about Infer.
Three states, three meanings, and only one of them is a fault.
Operational means requests are admitted and served inside the band.
Shedding means the fleet is at capacity and returning
429 — the designed behaviour, and the correct signal for a router to try
elsewhere for that request. Down is a real incident and is reported as
one, in plain language, with times.
- No queues. A request is admitted or refused on arrival. We do not hold requests to make an availability number look better; a held request is a slow request wearing a disguise.
- No 5xx for load. Overload is a
429. A5xxmeans something is genuinely broken on our side, so we keep that signal honest and rare. - Replica loss is absorbed internally. If a replica dies mid-stream, the gateway fails the session over and replays its context rather than surfacing an error. The caller sees a completed response, and the recomputed tokens are not billed as fresh prefill — our failure never inflates a bill.
- Retry semantics are boring on purpose.
429is retryable and carries no partial output. Nothing is half-written.
GLM-5.3-Flash
One model, run properly, is worth more than a catalogue. Figures below are the values we intend to publish on our OpenRouter listing; the listing is not yet live.
| Field | Value | Note |
|---|---|---|
| Model id | zai-org/GLM-5.3-Flash | Served by Infer |
| Quantization | fp8 | Weights and KV cache served in fp8 |
| Context length | 128,000 tokens (pending) | Native model context is 1,000,000 tokens; 128k is a conservative launch value awaiting sign-off, to protect the speed floor. Raised on measured evidence, not ambition. |
| Max output | 32,000 tokens | Default ceiling; configurable |
| Input price | $0.15 / M tokens | List price at launch |
| Output price | $0.50 / M tokens | List price at launch |
| Cache read | $0.03 / M tokens | Applies to prefix-cache hits |
| Streaming | SSE | Usage object returned in streaming and non-streaming responses |
| Tool calling | Supported | OpenAI-compatible tool definitions |
| Reasoning | Supported | Reasoning output parsed and returned |
| Multimodal input | Image input | Per the model specification |
| API surface | POST /v1/chat/completions | OpenAI-compatible; routed to us by OpenRouter |
| Region | United States (candidate, pending) | To be confirmed at submission; node placement is not yet finalized |
Not yet live. Infer is pre-launch. The listing goes ready only after staging load tests pass and measured throughput is in hand — we would rather list late than list a number we cannot hold.
Three mechanisms, no magic.
Everything that makes the speed floor achievable lives in the gateway. The GPUs are rented and interchangeable; the policy is not.
Session-affinity prefix caching
A conversation is hashed to a replica and stays there. Follow-up turns land on the machine that already holds their prefix state, so the shared prefix is reused instead of recomputed. Cached input bills at the cache-read rate, and the turn starts sooner because the prefill was not repeated.
Admission control
Two dimensions are measured continuously: decode-slot occupancy and uncached prefill
rate. Either one approaching its limit refuses the next request with a
429 on arrival. The refusal path is deliberately cheap — it touches no
database and adds no latency to the requests we do admit.
Constants from load tests
The floor, the band, and the absorbable prefill rate are not guesses; each is a constant fitted from load tests against the exact replica shape we deploy. When the hardware profile changes, the constants are re-measured before the advertised numbers move — never the other way round.
On provenance. Every number Infer publishes after launch will carry how it was measured and over what window. Until then, the figures on this page are labelled design targets, because that is exactly what they are.