The True Cost of LLM Latency

Time-to-first-token and total generation time are different products. Users experience the first; your infrastructure bill experiences the second. Almost every unproductive latency conversation I have sat through went wrong by conflating them.

Streaming is not an optimization

Streaming does not make generation faster. It makes waiting legible, which is a different and frequently more valuable thing. A response that takes eight seconds to finish but begins in four hundred milliseconds is experienced as responsive; the same response delivered atomically is experienced as broken.

Streaming also makes cancellation meaningful. A user who reads two sentences and navigates away has told you something useful, and a system that keeps generating is spending money on tokens nobody will read. Wiring client disconnects through to provider cancellation is one of the highest-leverage changes available, and it is usually a dozen lines.

Timeouts compose badly

A thirty-second client timeout in front of a thirty-second gateway timeout in front of a thirty-second provider timeout does not give you thirty seconds of tolerance. It gives you a system where the client gives up first and every layer beneath it keeps working on a request nobody is waiting for.

Under load, that orphaned work is exactly what prevents recovery.

// A deadline is absolute, not relative. Each layer gets whatever time is
// actually left, and fails fast when that is not enough.
export async function withDeadline<T>(
  deadlineAt: number,
  signal: AbortSignal,
  fn: (remainingMs: number, signal: AbortSignal) => Promise<T>,
): Promise<T> {
  const remaining = deadlineAt - Date.now();
  if (remaining <= 0) throw new Error("deadline exceeded");
 
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), remaining);
  signal.addEventListener("abort", () => controller.abort(), { once: true });
 
  try {
    return await fn(remaining, controller.signal);
  } finally {
    clearTimeout(timer);
  }
}

The rules that follow from this are short:

1. Set the budget once, at the edge, from the user-facing requirement.
2. Propagate the remaining time downward — never a fresh default.
3. Make each layer honour the deadline it was handed.
4. Cancel downstream work when the caller disconnects.

The cost side

Output tokens usually dominate the bill, and output length is the variable teams control least deliberately. A prompt that quietly encourages a preamble, a restatement of the question, and a closing summary can double generation time without changing the useful content by a word.

Before reaching for a cheaper model, measure how many tokens you are paying for that nobody reads. Trimming an answer format is free, reversible, and does not change the quality of the reasoning. Switching models is none of those things.

Where speculative decoding helps

Drafting with a small model and verifying with the large one works well for predictable output: structured formats, code completions, anything with strong local regularity. It helps much less on open-ended prose, where the draft diverges early and verification rejects most of what it proposed.

Knowing which of those describes your workload, before you invest in the machinery, is most of the work.