Streaming
Words made simple
A few useful meanings before you read the explanation.
- Streaming
- Sending an answer in small pieces as it is produced, rather than waiting for the complete response.
- API
- A way for one piece of software to ask another for data or an action. A claims API might return the current status of a claim.
- Token
- A small piece of text processed by a model. It may be a whole word, part of a word, or punctuation.
More words from this lesson
- JSON
- A common text format for exchanging data using named fields, lists, numbers and values. Example: {"status": "pending"}.
- Agent
- An application that uses a model to choose its next action within defined limits. It can use tools and continue through several steps.
- Persistence
- Saving information so it remains available after a request ends or a process restarts.
- Durable
- Able to retain progress and recover across interruptions. Durable workflows require appropriate storage and retry behavior.
- Authentication
- Checking who a user or service is, such as by verifying a login session.
Overview
- Streaming delivers response events as work happens instead of waiting for the entire response.
- Text events may contain small pieces called deltas.
- The browser can show progress and partial text while the server continues processing.
Why this matters
It improves perceived responsiveness and makes long requests easier to understand.
An everyday example
A restaurant tells you “your order is being prepared” before handing over the complete meal.
A technical example
A Node API forwards safe provider text events as server-sent events, or SSE, to a React client.
How it works
- Authenticate
- Open stream
- Emit typed progress/text events
- Handle disconnect and backpressure
- Emit final result or error
- Persist final status.
Architecture and flow
Provider events → backend event filter → SSE/streamed fetch → React incremental rendering.
The insurance assistant example
Show “Checking claim…” before displaying the verified status, without showing private tool arguments.
When to use it
Interactive chat, long answers, and visible agent progress.
When to avoid it
Do not execute partially streamed JSON or publish unvalidated high-risk advice token by token.
Interview answer
“Streaming reduces time until the user sees progress. The backend still owns authentication, cancellation, safe event filtering, and final validation.”
Benefits, trade-offs, and production considerations
- Time to first token differs from total latency.
- Streaming does not automatically reduce total cost or work.
- A browser disconnect should cancel optional generation, while a committed business operation needs durable status and reconciliation.
Can you explain it without the notes?
Streaming
Use your own words. Explain an example and one limitation before checking the explanation.
Check your explanation
Streaming delivers response events as work happens instead of waiting for the entire response. Text events may contain small pieces called deltas. The browser can show progress and partial text while the server continues processing.
Turn this concept into a clear answer.
Practice this conceptAdapted from 13_AI.MD · “Streaming”. Examples are study fragments; framework behavior can depend on your version.