Streaming AI chatbot

Overview

A streaming chatbot shows safe response pieces and progress as they arrive. Final completion remains a separate event.

Why this matters

Users see activity sooner during slow generation.

An everyday example

A support employee explains that they are checking records before finishing the answer.

A technical example

Stream typed SSE text, progress, done, and error events.

How it works

1. Authenticate
2. Establish provider stream
3. Project safe events
4. Handle backpressure/disconnect
5. Persist final status.

Architecture and flow

Provider stream → Node event adapter → fetch/SSE → incremental React state.

The insurance assistant example

Show claim lookup progress and then the verified explanation.

When to use it

Interactive tasks where waiting is noticeable.

When to avoid it

Do not expose raw tool events or execute partial JSON.

Interview answer

“I stream safe application events and explicitly distinguish completion, interruption, and failure.”

Benefits, trade-offs, and production considerations

Test proxy buffering, reconnect, abort, and errors after headers. Buffer sensitive output until validation when risk requires it.

// Server-side SDK illustration, not a complete HTTP route.
// Call only after authentication, quotas, validation, and context construction.
import OpenAI from "openai";
const client = new OpenAI();
async function* answerEvents(
  model: string, message: string, signal: AbortSignal,
): AsyncGenerator<{ type: "text" | "done"; value?: string }> {
  const stream = await client.responses.create({
    model, store: false, stream: true, max_output_tokens: 500,
    input: [
      { role: "system", content: "Explain supplied text. Do not invent account facts." },
      { role: "user", content: message },
    ],
  }, { signal });
  let completed = false;
  for await (const event of stream) {
    if (event.type === "response.output_text.delta") {
      yield { type: "text", value: event.delta };
    }
    if (event.type === "response.completed") completed = true;
    if (event.type === "response.failed" || event.type === "response.incomplete") {
      throw new Error("Generation did not complete");
    }
  }
  if (!completed) throw new Error("Stream ended without completion");
  yield { type: "done" };
}

• The HTTP adapter maps yielded values to its public stream format, handles refusals/provider errors according to product policy, applies backpressure, and converts errors into a safe terminal error event.
• The browser renders text as escaped content, never unchecked HTML.
• See official streaming documentation for the selected SDK's event types.