LLM calls with budgets, caching, and fallbacks.

Six fx:// functions

ai-auto classifies and routes; ai-fast, ai-code, ai-reason, ai-vision, ai-embed hit task-specific pools. All return the same typed envelope so callers do not branch on the underlying model.

Multi-provider ladder

Cloudflare Workers AI is the primary rung. Groq adds cheaper cross-provider fallbacks when configured. An on-prem ai-gateway gives a last-resort rung at $0 metering. Each rung has its own circuit breaker.

L0 exact + L1 semantic cache

Two-tier cache. Exact-match cache is the fast path. L1 stores bge-m3 embeddings with an HNSW cosine index; threshold auto-tunes toward a target false-positive rate so paraphrase hits survive while random collisions do not.

Weekly free quota + metered billing

Generous weekly allowance per plan with a daily cap as one-third of the weekly budget. Cache hits consume a fraction of quota, so effective throughput is much higher than the plan number. Over-quota returns a typed error, never a silent 429.

BYOK via Vault

Bring your own OpenAI / Groq / DeepSeek / Together / OpenRouter / Fireworks keys through Vault AI connections. BYOK bypasses platform quota but enforces the preset allowlist, per-connection RPM, and daily + monthly spend caps.

TokenFabric delta sync

Optional: re-send only the files that changed since the last call. Unchanged files are summarized by hash. Saves most tokens on codebase-aware prompts. Injection-guarded before cache write.

Managed inference

Workers AI primary + Groq cross-provider fallback + on-prem ai-gateway last resort. The router picks the rung per task from real outcome signals; circuit breakers skip rungs that fail.

• Five task pools (fast, code, reason, vision, embed) with curated rungs and live calibration
• Per-tenant cache namespace — global cross-tenant cache is opt-in only
• Fail-closed on quota exhaustion, missing creds, or rung exhaustion — no silent sandbox fallback
• Receipts minted per call through the platform substrate for replay and audit

Bring-your-own-key

Store OpenAI-compatible provider keys in Vault as AI connections. FlyAI calls them directly through the same router. BYOK bypasses platform quota but enforces preset allowlist, RPM, and daily/monthly caps.

• Six OpenAI-compatible presets: openai, groq, deepseek, together, openrouter, fireworks
• Bypass platform quota but enforce the preset allowlist and per-connection RPM
• Daily + monthly spend guardrails against ai_request_logs.cost_cents
• ZK-mode Vault connections are rejected at request time — fail closed

AIModelRouter unification

FlyAI and FlyMind share one inference substrate. Provider config, cache layer, telemetry, and embedding entry points are unified — providers, cache keys, and metrics all live in one place.

• Canonical config under the AIModelRouter substrate — FLY_AI_* settings continue to work as compatibility aliases
• Public error_code stays byte-identical across the rollout — internal_code is split for dashboards
• Canary routes specific fx:// functions one at a time (starting with ai-embed) before the global flip
• Rollback is a single flag flip — no rebuild required

Classify

For ai-auto: keyword heuristics pick the task (fast / code / reason / vision / embed). For the named functions, the task is fixed.

Claim quota

Atomic weekly-bucket + daily-cap claim from the live counters. DB-down or daily-cap reached → deny fail-closed (no silent 429, no double-charge).

L0 exact cache lookup

Content-addressed hash of the tenant + task + model + messages tuple. Hit returns the cached envelope with cached:true, cache:"exact" — provider is never called.

L1 semantic cache lookup

Optional. bge-m3 embedding of the prompt against HNSW cosine. Above threshold returns a semantic hit; below threshold falls through. Adaptive threshold tunes itself toward the target false-positive rate.

Screen for injection

Prompts and responses pass through the platform prompt-injection guard before cache write. Quarantined entries never poison future hits — the cache key is acked but the data is dropped.

Ladder dispatch

The router picks the best rung from real outcome signals — learning which provider is fastest for each task from live traffic. Otherwise priority order. One retry on 429 / 5xx with jittered backoff, then the next rung. Circuit breakers skip failing rungs entirely.

Provider call

Per-call timeout (default 30 s, max 60 s). max_tokens clamped to the configured ceiling. The selected provider runs the request and returns the response.

Cache + receipt + telemetry

Successful response is cached (L0 always; L1 if enabled). A typed, replayable execution record is minted. Telemetry increments the per-model + per-tenant series. Tenant usage is updated against the budget.

Auto-route chat

Classifies the prompt by task and routes to the right pool. Use when you do not care which model answers — just want the best answer for the prompt type.

Fast chat

Free-tier-first ladder: Llama 3.2 3B + 1B, then Qwen3 30B A3B MoE, then GPT-OSS 20B via Groq. 131K context on the front rungs.

Code generation

Qwen2.5 Coder 32B primary, GPT-OSS 120B (Groq + Cloudflare) as fallbacks, Llama 3.1 8B FP8 free rung. 32K–131K context across the ladder.

Reasoning

QwQ 32B primary, DeepSeek R1 Distill Qwen 32B and Llama 3.3 70B FP8 Fast fallbacks. All reasoning-tier — cost tracks with capability.

Vision

Llama 3.2 11B Vision primary, Llama 4 Scout 17B 16E fallback. 131K context both rungs.

Embeddings

bge-m3 primary (60K context), BGE Base and Large English as fallbacks. Cosine-indexed in pgvector for the semantic cache. Single-model semantics: embeddings never ladder-fail across providers.