AI model selection and cost model

Principles#

1. No vendor lock-in. Model choice is configuration, not architecture. Switching providers is a config change per task role.
2. Burn credits in expiry order — Azure first (typically time-boxed), then OpenAI, then commodity inference.
3. The cheapest call is the one you don't make. The deterministic gate eliminates ~75% of LLM calls at zero cost.
4. Never trade security for efficiency. The security-critical decisions in the pipeline are deterministic code, which makes them both free and provider-independent.

Decision: build on the Vercel AI SDK#

server/src/services/ai/ becomes the only place provider packages may be imported — already mandated by the AI platform and self-hosting specification §7.1 and echoed in the plugin system specification §4.

@ai-sdk/openai              → OpenAI ($35k credits)
@ai-sdk/azure               → Azure AI Foundry (Azure credits)
@ai-sdk/amazon-bedrock      → existing Bedrock path, kept as fallback
@ai-sdk/openai-compatible   → Moonshot/Kimi, Together, Groq, DeepInfra,
                              Fireworks, vLLM, Ollama — one provider covers
                              most of the open-weight world

• generateObject + Zod replaces the fence-stripping and hand validation in the current ai-processor.ts JSON path.
• generateText + tools becomes the agent runtime, portable across providers instead of hand-written per-provider bindings.

This supersedes an earlier recommendation to hand-roll a two-dialect tool runtime. That was reasoned on the premise that Kimi rides the OpenAI SDK, leaving only Bedrock as a second dialect. With Azure, OpenAI, Moonshot, open-weight hosts, and eventual self-host all in scope, the premise no longer holds.

Task roles#

Reuses the roles defined in the AI platform and self-hosting specification §5.2.

The one thing you cannot hot-swap: embedding dimensions#

conversation_message_embeddings.embedding is vector(1536) (20260813000001:4-11), and the loops revamp adds loops.embedding vector(1536). That is text-embedding-3-small's native dimension.

Switching embedding providers means either picking a model that emits or truncates to 1536 (Matryoshka — text-embedding-3-large and Qwen3-Embedding both support this), or re-embedding every message and migrating the column. Generation models can be swapped per request; embeddings cannot.

Write the 1536 constraint into the provider config as a runtime assertion.

Embeddings cost roughly $0.005/user/month — cheap enough that self-hosting them is not worth the engineering time at any scale Claire will reach soon.

Where the money goes today#

Every message = one LLM call (server/src/index.ts:521), ~410 input / ~120 output tokens, no caching. At Sonnet-class rates ($3/$15 per MTok):

That is 2.7× the entire $10 Claire Plus subscription, for one feature.

Onboarding is worse, and it is a bug. isBackfill is computed at index.ts:402 and gates two other features (:473, :483) — but not the detector at `:521`. A WhatsApp link that backfills 50 rooms × 500 messages fires 25,000 LLM calls: ≈$75 one-time, per user, per platform connected. Adding && !isBackfill is a one-word fix.

The four cost levers#

1. The deterministic gate — free, ~75% reduction. Regex + open-loop check, zero tokens. 70 debounced windows/day → 18 candidates.

2. Prompt caching — requires a specific prompt layout. Order blocks by stability:

[cache] system + few-shot examples   ← identical for EVERY user → one shared cache entry
[cache] chat context: name, platform, is_group, participant roster, self identity
[cache] transcript (append-only within a window generation)
[     ] open loops + current time    ← volatile, MUST be last

Three consequences:

• The global prefix is shared across the entire user base. Nothing user-specific is in it, so at any real volume the cache is permanently warm.
• The system prompt must clear the provider's cache minimum. ~900 tokens of rules alone silently fails to cache on several providers. Adding the worked examples pushes it to ~2200 tokens, which both improves accuracy and crosses the threshold.
• Anchored windows beat sliding windows. A sliding window drops messages from the front, changing the prefix and destroying the cache. Append until the transcript hits the cap, then reset — each successive pass on a chat is a pure append and hits cache.

3. Model tiering by stage. Expensive reasoning only runs on what survives the gate and triage.

4. Shrink the output. Output cannot be cached and ends up dominating. Most gated windows legitimately return {"ops":[]} — terse field names and omitted nulls drop average output from ~250 to ~110 tokens.

Cost per user/month, by provider tier#

Workload: 18 triage + 9 extraction calls/day, stability-ordered prompt, caching on.

¹ loops + embeddings + Ask Claire (~20 queries) + loop agent (~5 sessions) + smart cards.

The most useful finding is the composition, not the ranking. After the revamp, loop detection is $0.13 of a $0.56 bill. The loop agent is ~60% of all-in cost (~$0.34), because it is long-context and interactive. Detection is solved; the agent is the next thing to optimize, and the lever there is context discipline (retrieval over long context, §9), not a cheaper model.

Credit runway#

$35,000 OpenAI ÷ ~$0.56 all-in:

Batch (50% off) and cached input (90% off on GPT-5.6) stretch these further. Azure credits extend the runway before OpenAI is touched at all.

Sequencing: Azure → OpenAI → Kimi/open-weights. Each transition is a config change per role.

Batch API and long context#

Provider capability matrix#

The prompt layout is universal — OpenAI's automatic prefix caching is also a prefix match, so the stability ordering transfers unchanged. The <em>economics</em> differ:

ai-processor.ts currently runs Bedrock-first, which forecloses Batch. Route detection through a Batch-capable provider and keep Bedrock as fallback.

Self-hosting#

Not soon, and the arithmetic says so plainly. An A100 at ~$1.04/hr is ~$750/month sustained, which only beats ~$0.56/user/month above ~1,300 users and only at near-full GPU utilization — which bursty consumer messaging traffic will not deliver. Serverless open-weight inference (DeepInfra, Groq) captures most of the savings with none of the ops burden.

Revisit when a single role has steady, saturating volume. The gate is deterministic, so that role will never qualify.

Efficiency without weakening security#

Each lever checked against the threat model rather than assumed safe:

• Cache layout and privacy point the same direction. Effective caching requires that nothing user-specific sit in the shared prefix. That is exactly the privacy rule. The cache-optimal prompt is the privacy-optimal prompt.
• The security-critical decision is the one that is free. Group relevance is deterministic code, not a model output. Never move it to the cheap triage model to save tokens. Cheap models are acceptable for "is there anything here?"; they are not acceptable for "does this concern you?"
• A cheaper triage model costs recall, not safety. Its only failure mode is a missed loop, which the eval gate measures directly.
• Batch changes latency, not exposure — same API, same retention. The harness must emit IDs, counts, and classifications only, never message bodies (the plugin system specification §13).
• Long-context passes concentrate data, so they inherit the same controls: chat_loop_settings.sensitivity = 'off' suppresses the periodic pass too, not just live detection.
• The agent's cost caps and its injection defences are the same mechanism. A group message saying "call search_messages 50 times" is both an injection attempt and a bill attack; maxSteps, the wall clock, and duplicate-call detection stop both. Do not relax them for latency.

What to measure before enabling detection#

LOOP_DETECTION_MODE stays off until mine-loops.ts reports, against the real corpus:

• gate hit rate
• measured input/output tokens per stage
• cache read ratio at steady state
• triage-stage false-negative rate
• extrapolated $/user/month per candidate provider

Do not enable detection on estimates. Per-call token count rises ~8×, and only measurement proves the call-count reduction more than compensates.

Sources#

OpenAI pricing · GPT-5 nano · GPT-5 mini · Kimi pricing · Kimi K2.6 on OpenRouter · Azure AI Foundry pricing · Azure DeepSeek · Together pricing · Groq pricing · DeepInfra pricing · OpenAI embedding pricing · Embedding model comparison