Streaming Responses

How the chat route streams tokens while keeping auth, limits and credits in order.

Server side

AIClient.stream() calls your onChunk callback as text arrives. The chat route wraps it in a ReadableStream. This is the shape, with auth, rate limiting and credits left out for clarity; the real file keeps all of them.

// app/api/custom/chat/route.js (abridged)
import AIClient from "@/lib/ai-client";

export const runtime = "nodejs";

export async function POST(req) {
  const { messages } = await req.json();
  const ai = new AIClient("claude");
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      await ai.stream(messages, (chunk) => controller.enqueue(encoder.encode(chunk)));
      controller.close();
    },
  });

  return new Response(stream, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

Client side

components/features/AIChat.jsx reads the stream and appends to the last message. The response headers carry X-Conversation-Id and X-Credits-Remaining so the UI can update without a second request.

const res = await fetch("/api/custom/chat", {
  method: "POST",
  body: JSON.stringify({ messages, conversationId }),
});

const reader = res.body.getReader();
const decoder = new TextDecoder();
let reply = "";
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  reply += decoder.decode(value);
  setMessages((m) => [...m.slice(0, -1), { role: "assistant", content: reply }]);
}