Ask Claire v2: streaming, retrieval, and cost specification

Purpose and product boundary#

Ask Claire is the user-initiated research and assistance surface across connected conversations. It finds relevant evidence, answers with citations, and helps the person decide what to do next. It does not silently send messages, book meetings, edit external systems, or treat message text as authorization.

This specification covers the interactive Ask Claire tab, the conversation-scoped Ask Claire surface, and the shared server engine behind both. It extends the AI platform specification, the AI model and cost model, and the plugin system specification. Those documents remain authoritative for account billing, provider credentials, plugin permissions, approvals, and external execution.

Non-goals for v2: autonomous background agents, unrestricted MCP execution, model-generated UI, silently sending messages, and storing a full model transcript as the database source of truth.

Baseline audited before the v2 implementation#

The September 8 baseline was a persisted RAG chat. New messages were embedded withtext-embedding-3-small and stored in conversation_message_embeddings. A question launches lexical and vector retrieval in parallel, merges up to 12 individual message hits, adds the last six assistant turns and selected conversation instructions, calls OpenAI Chat Completions for JSON, validates cited indices and two navigation actions, then stores the user and assistant turns. The table records the gaps that drove v2; the shipped state and remaining launch work are summarized indocs/ASK_CLAIRE_V2_LAUNCH_READINESS.md.

Representative questions and capability bar#

Ask Claire succeeds only if it can answer the questions people naturally ask when they remember part of a conversation but not where it happened. The following questions are release-defining workloads, not aspirational examples. They become versioned golden queries in the evaluation corpus and must work across every enabled messaging platform.

{
  "intent": "identity_lookup | plan_recall | people_ranking | conversation_qa | action_request",
  "timeRange": { "start": "ISO-8601", "end": "ISO-8601", "timezone": "IANA" },
  "people": [{ "name": null, "descriptors": ["Russian"], "chatId": null }],
  "location": { "city": "Mexico City", "asOf": "ISO-8601" },
  "planStatus": ["proposed", "confirmed", "changed", "cancelled"],
  "ranking": { "kind": "relationship_strength", "limit": 12 },
  "requestedAction": { "type": "draft_outreach", "requiresApproval": true }
}

Target architecture#

Keep retrieval and persistence in Claire-owned services. The AI SDK is the provider and stream runtime, not the product architecture. Product code asks for a task profile and consumes Claire types; only apps/server/src/services/ai/ imports provider packages.

AI SDK adoption decision#

Model portability is real for text generation and embeddings only when Claire tests the required capability. Maintain a capability record per configured model: streaming, structured output, tool calling, maximum context, embedding dimensions, usage reporting, and simulated-stream fallback. Never assume that every provider implements the abstraction equally well.

Answer request lifecycle#

1. The client creates a UUID requestId and sends one request. A missing global thread is created inside this request.
2. The server authenticates the user, checks AI policy and per-user budget, validates scope ownership, and idempotently inserts the user turn plus an empty assistant turn with status=streaming.
3. The server immediately opens the stream and emits a sanitized retrieving state with the real persisted thread and turn IDs.
4. The context router chooses deterministic sources. “Catch me up” reads a recent window; “what is unresolved?” reads Loops; scoped factual search uses hybrid retrieval. This choice does not require another model.
5. Retrieval creates a bounded evidence pack. If no candidate passes the relevance floor, emit a deterministic no-evidence answer and spend no generation tokens.
6. The budget service reserves the maximum permitted request cost. The model router selects an eligible model profile and provider candidate.
7. streamText streams the grounded answer. Citations are stable source IDs, not array positions that can change between steps.
8. On completion, validate citation markers and proposal parts, persist the final answer and usage atomically, settle the reservation, and emit finish.
9. On provider failure before the first text delta, try the next eligible provider. After text has been shown, do not silently replay through another model; mark the turn failed/partial and offer an idempotent retry.

Streaming contract#

Add content negotiation to the existing message endpoints. Accept: text/event-streamselects the v2 AI SDK UI stream; Accept: application/json preserves the existing response during rollout. Do not create separate business logic for the two transports.

type AskClaireRequest = {
  requestId: string;             // client UUID, unique per user
  threadId?: string;             // omitted to create a global thread
  question: string;
  scope:
    | { mode: 'global' }
    | { mode: 'preferred'; chatIds: string[] } // 1..5
    | { mode: 'strict_chat'; chatId: string };
};

type ClaireStreamData = {
  status: { phase: 'retrieving' | 'answering' | 'saving' };
  thread: { threadId: string; userTurnId: string; assistantTurnId: string };
  sources: { citations: AssistantCitation[]; indexStatus: AssistantIndexStatus };
  proposal: { proposalId: string; capabilityId: string; requiresApproval: boolean };
  finish: { finishReason: string; provider: string; model: string };
  error: { code: string; retryable: boolean };
};

Emit source metadata before answer text so citations can render without waiting for the final token. The text itself should be normal readable Markdown with stable citation marks such as [S1]. Do not stream serialized JSON into the answer bubble. Use custom data parts for Claire metadata and future proposals.

Provider and model routing#

Generalize the existing loop-only provider registry into Claire's shared AI service. Ask Claire must request a profile, not a vendor model string. Provider credentials and model IDs remain configuration.

A provider fallback is eligible only when it matches the profile's capabilities and data policy. Failover may happen before the first visible delta. Mid-stream failure is surfaced honestly because concatenating two providers' answers can duplicate text, change claims, and invalidate citations.

Retrieval v2#

Cost controls#

Measure cost from provider-reported usage and the versioned price book in integer micro-USD. Do not hard-code a permanent dollar estimate in this document. For requestr:

estimated_max_cost(r) =
  query_embedding_tokens × embedding_rate
  + uncached_input_tokens × input_rate
  + cache_read_tokens × cache_read_rate
  + max_output_tokens × output_rate
  + permitted_additional_steps × step_reserve

settled_cost(r) = provider-reported usage × price_book(version)

Message indexing specification#

• Replace process-local fire-and-forget backfill with a durable per-user job and bounded provider concurrency. Message ingestion must never wait for embedding.
• Use embedMany; split by provider maximum and token budget, then bulk upsert vectors in the same order.
• Select rows whose embedding is missing or whose stored content_hash differs. Content edits must become searchable.
• Delete or exclude vectors when messages are soft-deleted, and keep index counts scoped to eligible live text rows.
• Maintain index progress incrementally. Do not run exact counts over messages and embeddings during every answer request.
• Store embedding_model, dimensions, and embedding_version. A model/dimension change creates a parallel versioned index and controlled cutover, not mixed vectors.
• Retry transient provider failures with jitter and a ceiling; expose permanent failures without blocking lexical search.

Data model and API changes#

Persist the user turn and assistant placeholder in one database function. Finalize the assistant turn, update the thread title/timestamp, and settle the usage record atomically where practical. Return the updated thread in the terminal stream data so the client does not reload the entire thread list after every answer. Thread history uses keyset pagination; generation reads only the bounded recent slice.

Path to plugins and real actions#

Keep open_conversation as trusted Claire navigation. Rename the current calendar affordance to “Open calendar” so it does not imply that anything was created. When plugins land, Ask Claire receives two classes of tools:

The streamed proposal part contains only a Claire proposalId and display metadata. Approval reloads the proposal from the server, verifies the payload hash and destination, then queues idempotent execution and writes a receipt. External writes and invitations always follow the plugin specification even if the AI SDK marks a tool call approved.

Default Ask Claire remains one-step RAG. Enable tool calling only when an installed, granted capability is relevant or the question requires a structured Claire read tool. Cap global Ask Claire at three model steps, 20 seconds total, five seconds per read tool, 4KB per tool result, and one proposal per user intent. A model denial response must not cause it to retry the same proposal.

Privacy and security requirements#

• Every service-key database query includes user_id; every chat and thread scope is ownership-checked server-side.
• Telemetry sets recordInputs: false and recordOutputs: false. Metadata may include request ID, route, provider, model, token counts, timings, source count, cache status, and error class—never message text, question text, embeddings, tool inputs, or provider credentials.
• Only citations selected for the answer leave the server in source parts. Retrieval candidates that were not used are not exposed to plugins.
• Remote provider and plugin data handling follows the user's configured consent and the plugin manifest. BYOK is not described as local.
• Sensitive inferred person attributes are ephemeral by default. Persisting them requires the applicable memory policy, purpose limitation, retention, correction, deletion, and source provenance; they are never exposed to a plugin merely because they appeared in retrieval.
• Prompt injection fixtures must prove that message text cannot expand scope, select a hidden destination, grant a capability, lower approval, or cause direct execution.
• Error events expose stable safe codes; raw provider errors remain in redacted server logs.

Observability and cost accounting#

Record one trace and one usage event per answer with privacy-safe spans:

assistant.request
  ├─ thread.load
  ├─ context.route
  ├─ retrieval.lexical
  ├─ retrieval.embedding
  ├─ retrieval.vector
  ├─ evidence.assemble
  ├─ model.first_attempt | model.fallback
  │    ├─ time_to_first_token
  │    └─ stream_duration
  └─ turn.finalize

Dashboards break down p50/p95 time to first token, completion latency, input/output/cache tokens, estimated and settled micro-USD, no-call rate, fallback rate, cancellation rate, stream failure rate, retrieval route, source count, and model profile. Alerts use error and budget rates, not conversation content.

Evaluation and test plan#

Reuse the Lucas two-device context-token scenario for real bridge grounding. Add a deterministic AI SDK mock stream so UI and server tests require no provider key and do not depend on token timing.

Release acceptance gates#

Numeric latency and quality gates are launch targets, not claims about the current system. Capture the current baseline before PR 1 and revise targets only from measured staging and device evidence.

Delivery plan#

1. PR 0 · Now

   Correctness and baseline

   Fix conversation-thread access, add global/chat integration tests, and record current latency, tokens, request counts, and cost.

2. PR 1

   Shared AI runtime

   Move generation and embeddings behind services/ai with task profiles, provider contracts, timeouts, and usage normalization.

3. PR 2

   Hot path and index

   Add recent-turn queries, incremental index state, indexed full-text search, embedMany, atomic finalization, and idempotency.

4. PR 3

   Server streaming

   Ship AI SDK UI streams over Express with typed parts, cancellation, finalization, and a legacy JSON fallback.

5. PR 4

   Client streaming

   Add the shared Expo and Electron stream adapter, incremental rendering, Stop, recovery, and accessibility.

6. PR 5

   Personal-memory retrieval v2

   Add structured query compilation, relative dates, message windows, person facts, relationship metrics, plan candidates, and golden-query evals.

7. PR 6 · After plugins

   Durable action proposals

   Expose installed read and propose tools through the plugin policy engine without granting the model direct write access.

Roll out by account: internal fixtures → Lucas two-device account → 5% → 25% → 100%. Keep the JSON path and previous provider profile available for one release. Automatically roll back on grounding, cost, stream-error, or latency gate regression.

Primary implementation map#

• apps/server/src/services/conversation-assistant.ts — split orchestration, retrieval, persistence, and transport responsibilities.
• apps/server/src/services/ai/ — shared model profiles, provider capabilities, streamed generation, embeddings, usage, and test mocks.
• apps/server/src/routes/ai.ts — content negotiation, stream lifecycle, abort, and legacy JSON fallback.
• supabase/migrations/ — turn lifecycle/idempotency, embedding versioning, full-text index, and atomic database functions.
• apps/client/services/conversationAssistant.ts — typed stream transport and legacy fallback.
• apps/client/hooks/useAssistantStream.ts — shared mobile, in-chat, and Electron state adapter.
• apps/client/components/claire/ — shared answer, source, proposal, status, stop, and error parts.

External implementation references#

• AI SDK streamText reference
• AI SDK Express streaming
• AI SDK Expo streaming with expo/fetch
• AI SDK UI stream protocol
• Provider and model management
• AI SDK embedMany reference
• Tool calling and approval parts
• Telemetry input/output recording controls
• Deterministic model testing