Streaming chat route under auth

The starter boots, but the chat rail on /invoices is dead: the textarea is disabled, and POST /api/chat answers every request with a 501. This lesson brings that endpoint to life. As a member of org-acme, you’ll type a question into a temporary chat box and watch a text answer stream back; anyone who isn’t an authenticated org member gets refused before the model runs.

The slice is deliberately narrow: a streaming answer with no tools yet, gated by auth, with the agentic loop’s cost ceilings bolted on before there’s anything expensive to run. Tools, per-user budgets, and the real UI come in lessons 3 through 5, but all of it depends on this capped, authenticated route existing first. Since the real chat UI is the last lesson, you’ll stand up a throwaway box here that renders whatever streams back as plain text.

The result is behavioral, not visual. Type tell me a joke about invoices and watch text appear in the box. Flip the inspector’s BYPASS_AUTHED_ROUTE flag and the same request comes back 401. Open /inspector after a conversation and find exactly one llm.finish row in the LLM audit-events tail.

Your mission

You’re wiring POST /api/chat to stream an LLM answer, and the order is the point: cap and wrap first, then add capability. The guardrails go in before the thing they guard exists, so capability can never be added without them. Three guardrails land in this lesson while the route still has nothing to call: the auth boundary, the step cap, and the output-token ceiling.

Start with the boundary. The route is wrapped in authedRoute('member', …), the Request/Response sibling of authedAction from The authedRoute twin. Calling streamText from a bare POST handler is the canonical LLM bug: it answers any caller on the internet, burns tokens on their behalf, and has no orgId to scope the answer. Streaming forces a route handler rather than a Server Action, since actions can’t stream, which is why authedRoute exists.

Then the loop cap. streamText runs an agentic loop: once tools exist, the model can call one, read the result, and decide to call another, round and round. stopWhen: stepCountIs(5) caps that at five steps. Set it now, even though no tool fires yet, because the moment a tool lands the runaway-loop window opens, and a cap you add later is one you forgot to add under pressure. The SDK default is stepCountIs(20), far too loose for a surface that bills a token budget per user. Alongside it, maxOutputTokens: 1024 caps a single answer’s length; a missing output cap is as serious as a missing auth check.

The system prompt is the third piece, and it is a controller, not a greeting. It does four jobs: scope answers to the acting org by name, force the model to ground numeric claims in a tool call rather than invent them, refuse questions about any other org’s data, and define what to do when a tool returns an error. One subtlety: the org name is templated into the prompt string, but the user’s typed question never is, it stays in the messages array. That separation is the prompt-injection boundary: trusted instructions live in the system prompt you control; untrusted user input stays in the message stream, where the model treats it as data to answer, not as instructions to obey.

Two seams from the Vercel AI SDK v5 appear once here, both owned by streamText, generateText, and the route-handler seam. convertToModelMessages translates the rich UIMessage[] the client renders into the flat ModelMessage[] the model expects on the wire; forgetting it is the classic first-week-with-v5 bug. And toUIMessageStreamResponse() makes the response a stream the client’s useChat can parse, where the lookalike toTextStreamResponse() speaks a protocol useChat won’t understand.

One trade to name: the input schema accepts z.array(z.unknown()) for messages rather than validating the full UIMessage shape, because convertToModelMessages does the real structural validation downstream.

Finally, the audit lineage. Every finished conversation writes one row, the same one-row-per-event discipline from The append-only audit log. This is a different table from that chapter’s auditLogs — the LLM-specific events log — so the writer is not pushAudit; against real Postgres each write is a single INSERT into llm_audit_events, and here a single array push stands in for it.

A few things are deliberately out of scope: the getInvoiceStats tool is the next lesson, the withLlmQuota reservation that will wrap this route is the lesson after, and the real chat client is this chapter’s last lesson. The one error-handling rule for now: onError returns a sanitized log and never leaks a raw error to the client.

Typing a question such as “tell me a joke about invoices” streams a text answer back into the temporary chat box.

untested

With BYPASS_AUTHED_ROUTE on, POST /api/chat is refused with 401 and the model never runs.

untested

A question that asks for another organization’s data is refused in the answer text (the system prompt as controller).

untested

Every completed conversation writes exactly one llm.finish row (with finishReason: 'stop') to the LLM audit-events tail, scoped to the acting org.

untested

Coding time

Implement the four files this slice touches against the brief above — src/lib/llm/prompts.ts, src/lib/llm/audit.ts, src/app/api/chat/route.ts, and a throwaway smoke-test src/app/(app)/invoices/invoice-chat.tsx — then open the walkthrough to compare.

Reference solution and walkthrough

We build in dependency order: the prompt and audit writers first (the route imports both), then the route, then the temporary client that exercises it.

The system prompt

src/lib/llm/prompts.ts ships as a one-line placeholder. Replace it with the four-rule controller:

import 'server-only';

// The system prompt is the controller. It templates the org display name (never
// user input — that stays in `messages`, the prompt-injection rule) and carries
// three load-bearing rules: tools are the only doorway to numbers, cross-org
// questions are refused, and a tool `{ error }` is read back as a graceful note.
export const invoiceQAPrompt = (ctx: { orgName: string }): string =>
  [
    `You answer questions about invoices for ${ctx.orgName} only.`,
    'Always call getInvoiceStats before stating any numeric fact about invoices — never guess counts, totals, or status breakdowns from memory.',
    `Refuse questions about any other organization's invoices; you can only see ${ctx.orgName}'s data.`,
    'If getInvoiceStats returns an { error }, tell the user the stats are unavailable right now and to try again — do not invent numbers.',
  ].join('\n');

Two rules name getInvoiceStats, a tool that doesn’t exist until the next lesson — that’s intentional. The prompt describes the contract you’re about to build the model against, so the moment the tool lands the model already knows to reach for it. import 'server-only' keeps the prompt out of any client bundle, turning a bundling mistake into a build error.

The single line carrying the most weight is ${ctx.orgName}, and equally what is absent. The org name is trusted data, so it gets interpolated; the user’s question is untrusted, so it never touches this string and flows separately through messages — the prompt-injection boundary in one decision.

The audit writers

src/lib/llm/audit.ts ships with both functions stubbed to no-ops and the argument types already in place. Fill in the two pushes:

import 'server-only';

import { pushLlmAuditEvent } from '@/server/store';

type StepArgs = {
  userId: string;
  orgId: string;
  finishReason?: string;
  usage?: unknown;
  toolCalls?: unknown;
};

type FinishArgs = {
  userId: string;
  orgId: string;
  finishReason?: string;
  usage?: unknown;
};

// One append-only row per agentic step. The SQL lineage's "bounded one-row
// transaction" is a single push here.
export const writeLlmStepEvent = async (args: StepArgs): Promise<void> => {
  pushLlmAuditEvent({
    userId: args.userId,
    orgId: args.orgId,
    event: 'llm.step',
    payload: {
      finishReason: args.finishReason,
      usage: args.usage,
      toolCalls: args.toolCalls,
    },
  });
};

// One append-only row per finished conversation.
export const writeLlmFinishEvent = async (args: FinishArgs): Promise<void> => {
  pushLlmAuditEvent({
    userId: args.userId,
    orgId: args.orgId,
    event: 'llm.finish',
    payload: {
      finishReason: args.finishReason,
      usage: args.usage,
    },
  });
};

Each writer is a single append-only pushLlmAuditEvent call, the in-memory stand-in for one INSERT into llm_audit_events. The event discriminant ('llm.step' vs 'llm.finish') and a jsonb-shaped payload are the whole row. This is the same one-row-per-event discipline from The append-only audit log, pointed at a separate table: keeping the LLM events log apart from auditLogs keeps the lifecycle audit (who archived which invoice) from tangling with the LLM audit (what the model did per step).

The route only calls writeLlmFinishEvent; writeLlmStepEvent sits unused until the next lesson wires the per-step audit, but both writers live in one file, so fill them in together.

The route handler

src/app/api/chat/route.ts ships as a 501 stub. Replace it with the wrapped, capped streaming handler:

import { convertToModelMessages, stepCountIs, streamText } from 'ai';
import { z } from 'zod';
import { authedRoute } from '@/lib/authed-route';
import { writeLlmFinishEvent } from '@/lib/llm/audit';
import { chatModel } from '@/lib/llm/models';
import { invoiceQAPrompt } from '@/lib/llm/prompts';
import type { InvoiceUIMessage } from '@/lib/llm/tools';

export const POST = authedRoute(
  'member',
  z.strictObject({ messages: z.array(z.unknown()) }),
  async (input, ctx) => {
    const org = await ctx.db.query.organization.findFirst({
      where: (o) => o.id === ctx.orgId,
    });
    const orgName = org?.name ?? 'your organization';

    const result = streamText({
      model: chatModel,
      system: invoiceQAPrompt({ orgName }),
      messages: convertToModelMessages(input.messages as InvoiceUIMessage[]),
      stopWhen: stepCountIs(5),
      maxOutputTokens: 1024,
      onFinish: ({ usage, finishReason }) =>
        writeLlmFinishEvent({
          userId: ctx.userId,
          orgId: ctx.orgId,
          finishReason,
          usage,
        }),
      onError: ({ error }) => {
        console.error('[chat] stream error', { code: 'stream_error' });
        void error;
      },
    });

    return result.toUIMessageStreamResponse();
  },
);

The auth boundary. authedRoute('member', …) is a route handler (not a Server Action — actions can’t stream), wrapped so the model only runs for an authenticated org member. ctx is flat: ctx.userId / ctx.orgId, never ctx.user.id. The schema accepts z.array(z.unknown()) because convertToModelMessages does the real structural validation downstream.

import { convertToModelMessages, stepCountIs, streamText } from 'ai';
import { z } from 'zod';
import { authedRoute } from '@/lib/authed-route';
import { writeLlmFinishEvent } from '@/lib/llm/audit';
import { chatModel } from '@/lib/llm/models';
import { invoiceQAPrompt } from '@/lib/llm/prompts';
import type { InvoiceUIMessage } from '@/lib/llm/tools';

export const POST = authedRoute(
  'member',
  z.strictObject({ messages: z.array(z.unknown()) }),
  async (input, ctx) => {
    const org = await ctx.db.query.organization.findFirst({
      where: (o) => o.id === ctx.orgId,
    });
    const orgName = org?.name ?? 'your organization';

    const result = streamText({
      model: chatModel,
      system: invoiceQAPrompt({ orgName }),
      messages: convertToModelMessages(input.messages as InvoiceUIMessage[]),
      stopWhen: stepCountIs(5),
      maxOutputTokens: 1024,
      onFinish: ({ usage, finishReason }) =>
        writeLlmFinishEvent({
          userId: ctx.userId,
          orgId: ctx.orgId,
          finishReason,
          usage,
        }),
      onError: ({ error }) => {
        console.error('[chat] stream error', { code: 'stream_error' });
        void error;
      },
    });

    return result.toUIMessageStreamResponse();
  },
);

The context carries ids, not the org’s display name — so the route fetches it. ctx.db.query.organization.findFirst is a store facade shaped exactly like Drizzle’s db.query.* read. The ?? 'your organization' fallback keeps the prompt sensible if the lookup ever misses.

import { convertToModelMessages, stepCountIs, streamText } from 'ai';
import { z } from 'zod';
import { authedRoute } from '@/lib/authed-route';
import { writeLlmFinishEvent } from '@/lib/llm/audit';
import { chatModel } from '@/lib/llm/models';
import { invoiceQAPrompt } from '@/lib/llm/prompts';
import type { InvoiceUIMessage } from '@/lib/llm/tools';

export const POST = authedRoute(
  'member',
  z.strictObject({ messages: z.array(z.unknown()) }),
  async (input, ctx) => {
    const org = await ctx.db.query.organization.findFirst({
      where: (o) => o.id === ctx.orgId,
    });
    const orgName = org?.name ?? 'your organization';

    const result = streamText({
      model: chatModel,
      system: invoiceQAPrompt({ orgName }),
      messages: convertToModelMessages(input.messages as InvoiceUIMessage[]),
      stopWhen: stepCountIs(5),
      maxOutputTokens: 1024,
      onFinish: ({ usage, finishReason }) =>
        writeLlmFinishEvent({
          userId: ctx.userId,
          orgId: ctx.orgId,
          finishReason,
          usage,
        }),
      onError: ({ error }) => {
        console.error('[chat] stream error', { code: 'stream_error' });
        void error;
      },
    });

    return result.toUIMessageStreamResponse();
  },
);

The two cost caps are non-negotiable. stopWhen: stepCountIs(5) can’t fire this lesson with no tools to call, but it’s set first on purpose, so the next lesson adds a tool into an already-capped loop. maxOutputTokens: 1024 ceilings a single answer. convertToModelMessages translates the UIMessage shape to the model’s wire shape.

import { convertToModelMessages, stepCountIs, streamText } from 'ai';
import { z } from 'zod';
import { authedRoute } from '@/lib/authed-route';
import { writeLlmFinishEvent } from '@/lib/llm/audit';
import { chatModel } from '@/lib/llm/models';
import { invoiceQAPrompt } from '@/lib/llm/prompts';
import type { InvoiceUIMessage } from '@/lib/llm/tools';

export const POST = authedRoute(
  'member',
  z.strictObject({ messages: z.array(z.unknown()) }),
  async (input, ctx) => {
    const org = await ctx.db.query.organization.findFirst({
      where: (o) => o.id === ctx.orgId,
    });
    const orgName = org?.name ?? 'your organization';

    const result = streamText({
      model: chatModel,
      system: invoiceQAPrompt({ orgName }),
      messages: convertToModelMessages(input.messages as InvoiceUIMessage[]),
      stopWhen: stepCountIs(5),
      maxOutputTokens: 1024,
      onFinish: ({ usage, finishReason }) =>
        writeLlmFinishEvent({
          userId: ctx.userId,
          orgId: ctx.orgId,
          finishReason,
          usage,
        }),
      onError: ({ error }) => {
        console.error('[chat] stream error', { code: 'stream_error' });
        void error;
      },
    });

    return result.toUIMessageStreamResponse();
  },
);

onFinish writes the single llm.finish row via writeLlmFinishEvent, scoped to ctx.orgId. onError logs a sanitized, code-only line and drops the raw error with void error — nothing about a model failure leaks to the client.

import { convertToModelMessages, stepCountIs, streamText } from 'ai';
import { z } from 'zod';
import { authedRoute } from '@/lib/authed-route';
import { writeLlmFinishEvent } from '@/lib/llm/audit';
import { chatModel } from '@/lib/llm/models';
import { invoiceQAPrompt } from '@/lib/llm/prompts';
import type { InvoiceUIMessage } from '@/lib/llm/tools';

export const POST = authedRoute(
  'member',
  z.strictObject({ messages: z.array(z.unknown()) }),
  async (input, ctx) => {
    const org = await ctx.db.query.organization.findFirst({
      where: (o) => o.id === ctx.orgId,
    });
    const orgName = org?.name ?? 'your organization';

    const result = streamText({
      model: chatModel,
      system: invoiceQAPrompt({ orgName }),
      messages: convertToModelMessages(input.messages as InvoiceUIMessage[]),
      stopWhen: stepCountIs(5),
      maxOutputTokens: 1024,
      onFinish: ({ usage, finishReason }) =>
        writeLlmFinishEvent({
          userId: ctx.userId,
          orgId: ctx.orgId,
          finishReason,
          usage,
        }),
      onError: ({ error }) => {
        console.error('[chat] stream error', { code: 'stream_error' });
        void error;
      },
    });

    return result.toUIMessageStreamResponse();
  },
);

The v5 stream-response shape the client’s useChat can parse. Its lookalike toTextStreamResponse() speaks a different protocol useChat can’t read — the stream still flows, but your chat box stays empty with no error to explain why.

← Previous 1 / 1 Next →

Caution

The route.ts committed in the repo is the finished version, carrying more than this slice: a tools argument on streamText, an onStepFinish callback, and a withLlmQuota wrapper around the export. Those belong to the next two lessons. Build only what’s shown above for now — copy the committed file and your chat fails to typecheck against this lesson’s empty tool stub.

One detail the annotations don’t cover: the schema lets messages through as unknown[], so convertToModelMessages(input.messages as InvoiceUIMessage[]) casts to assert the shape the converter then validates — the two halves of the validation trade. InvoiceUIMessage imports cleanly from tools.ts even though its tool map is still an empty stub, so this typechecks today.

The throwaway smoke-test client

You need a way to drive the route before the real chat UI exists. invoice-chat.tsx ships as a disabled shell with the textarea greyed out. Replace it with a minimal useChat client that renders whatever streams back as plain text:

'use client';

import { useChat } from '@ai-sdk/react';
import { DefaultChatTransport } from 'ai';
import { type FormEvent, useState } from 'react';
import { Button } from '@/components/ui/button';
import type { InvoiceUIMessage } from '@/lib/llm/tools';

type InvoiceChatProps = {
  orgName: string;
};

// Throwaway smoke-test client — just enough to drive POST /api/chat and prove
// text streams back. It renders only text parts, as raw text. The real
// parts-rendering chat (text bubbles + the stats card) replaces this in the
// last lesson of the chapter.
export const InvoiceChat = ({ orgName }: InvoiceChatProps) => {
  const { messages, sendMessage } = useChat<InvoiceUIMessage>({
    transport: new DefaultChatTransport({ api: '/api/chat' }),
  });
  const [input, setInput] = useState('');

  const onSubmit = (event: FormEvent<HTMLFormElement>) => {
    event.preventDefault();
    if (input.trim() === '') {
      return;
    }
    sendMessage({ text: input });
    setInput('');
  };

  return (
    <div
      data-testid="invoice-chat"
      className="flex h-full flex-col gap-3 rounded-lg border p-4"
    >
      <div>
        <h2 className="text-sm font-medium">Ask your invoices</h2>
        <p className="text-xs text-muted-foreground">
          Smoke test — {orgName}&apos;s invoices.
        </p>
      </div>

      <div className="flex-1 space-y-3 overflow-y-auto text-sm">
        {messages.map((message) => (
          <div key={message.id} className="space-y-1">
            <span className="text-xs font-medium text-muted-foreground">
              {message.role === 'user' ? 'You' : 'Assistant'}
            </span>
            {message.parts.map((part, index) =>
              part.type === 'text' ? (
                <p key={`${message.id}-${index}`} className="whitespace-pre-wrap">
                  {part.text}
                </p>
              ) : null,
            )}
          </div>
        ))}
      </div>

      <form onSubmit={onSubmit} className="flex items-end gap-2">
        <textarea
          data-testid="chat-input"
          value={input}
          onChange={(event) => setInput(event.target.value)}
          rows={2}
          placeholder="Tell me a joke about invoices"
          className="flex-1 resize-none rounded-md border bg-background px-2 py-1.5 text-sm"
        />
        <Button type="submit" size="sm" data-testid="chat-send">
          Send
        </Button>
      </form>
    </div>
  );
};

This is scaffolding: the last lesson of the chapter throws it away for the typed client that renders tool-part cards across their lifecycle states. So it does the bare minimum — send a message, render the text parts that come back.

Three v5 details to register. The endpoint goes on the transport (new DefaultChatTransport({ api: '/api/chat' })), not as a top-level api option, which @ai-sdk/react@2 removed. The input state is yours to manage with useState(''), because v5’s useChat no longer manages it. And sendMessage({ text: input }) appends a user message and kicks off the stream. The render walks message.parts and keeps only text parts, dropping every other type (tool invocations, once they exist) with : null. Why messages are arrays of parts rather than one string is owned by useChat, useObject, and the parts array.

streamText — API reference

ai-sdk.dev

Every option on the call you're writing: stopWhen, maxOutputTokens, onFinish, onError, and toUIMessageStreamResponse.

Tool calling — multi-step (stopWhen)

ai-sdk.dev

How the agentic loop steps, why the default is stepCountIs(20), and the cap you set first this lesson.

Chatbot — useChat and message.parts

ai-sdk.dev

The v5 client surface your smoke-test driver uses: DefaultChatTransport, sendMessage, and rendering text parts.

Moment of truth

This project ships no per-lesson test suite — lesson-verification/ is a harness slot, so the project itself is the assessment. Your automated check is pnpm verify, which runs Biome’s CI lint, tsc --noEmit, and a next build with SKIP_ENV_VALIDATION=true:

pnpm verify

A clean run confirms the shape: the slice typechecks and the production build succeeds. No real key is needed, since the build makes no live model call — it stays green without AI_GATEWAY_API_KEY. The behavior you confirm by hand.

The four checks below stream from a real model, so they need AI_GATEWAY_API_KEY in your .env (copy .env.example, paste the key from the Vercel AI Gateway dashboard). You start as member-A, the default identity, so begin at /invoices; the inspector at /inspector is where you flip flags and read the audit tail.

Acting as member-A, typing “tell me a joke about invoices” streams a text answer into the smoke-test box.

untested

With the inspector’s BYPASS_AUTHED_ROUTE flag on, POST /api/chat returns 401 from authedRoute and the model never ran — then revert the flag.

untested

Asking for another organization’s data (for example, “how many invoices does Globex have?”) is refused in the answer text, because the system prompt is the controller.

untested

After one conversation, the inspector’s LLM audit-events tail shows exactly one llm.finish row with finishReason: 'stop', scoped to org-acme.

untested