Streaming

Server-sent events with usage on the final chunk. Handle partial lines; chunks do not align to events.

Set stream: true. The response is text/event-stream with data: {...} chunks and a final data: [DONE].

Consuming the stream

const res = await fetch("https://api.relayrouter.org/v1/chat/completions", {
  method: "POST",
  headers: {
    "content-type": "application/json",
    authorization: `Bearer ${process.env.RELAY_API_KEY}`,
  },
  body: JSON.stringify({ model, messages, stream: true }),
});

if (!res.ok) throw new Error(await res.text());

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";

for (;;) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data:")) continue;
    const payload = line.slice(5).trim();
    if (payload === "[DONE]") return;

    const chunk = JSON.parse(payload);
    const delta = chunk.choices?.[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
    if (chunk.usage) console.log("usage", chunk.usage);
  }
}

Stream semantics

The final chunk may carry usage because RelayRouter requests stream_options.include_usage.
If the client disconnects mid-stream, usage generated up to that point is still billed.
If the upstream fails mid-stream, the stream terminates and usage parsed so far is recorded.
Chunks are not guaranteed to align with event boundaries. Always keep the trailing partial line in a buffer.

Do not retry blindly

A client-side timeout or network retry can double-bill a stream that was already served. There is no Idempotency-Key support yet.

PreviousChat completions NextReliability