Prompt Caching

Prompt caching (reusing previously processed prompt tokens instead of reprocessing them) reduces cost and latency on repeated requests. By default, Arbytra handles cache optimization automatically. For fine-grained control, you can specify cache control manually.

Prerequisites

An Arbytra API key
Python 3.10+ with the OpenAI SDK (pip install openai) or the arbytra SDK (pip install arbytra)
- OR Node.js 18+ with the OpenAI SDK (npm install openai) or @arbytra/sdk (npm install @arbytra/sdk)

How it works

Arbytra optimizes caching for each provider when your request includes reusable prompt content. On subsequent requests sharing the same prompt prefix, the provider serves cached tokens at reduced cost and lower latency. Arbytra accounts for each provider’s caching economics (token thresholds, discount depths, and read/write prices) when choosing where to route. Over time, the system learns your usage patterns to improve estimation accuracy. Create separate workspaces for different use cases to get better predictions. Arbytra is a zero data retention proxy. Your prompts, responses, and content are never read, logged, or stored. Pattern calibration uses usage metadata only. Read the Privacy Policy for details.

Send a cached request

Send a request with a reusable system prompt:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1",
)

response = client.chat.completions.create(
    model="claude-sonnet-4-20250514",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant..."},
        {"role": "user", "content": "Explain async/await in Python."},
    ],
)

usage = response.usage
if hasattr(usage, "prompt_tokens_details") and usage.prompt_tokens_details:
    cached = getattr(usage.prompt_tokens_details, "cached_tokens", 0)
    print(f"Cached tokens: {cached}")
print(f"Total prompt tokens: {usage.prompt_tokens}")

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "claude-sonnet-4-20250514",
    messages: [
        { role: "system", content: "You are a helpful coding assistant..." },
        { role: "user", content: "Explain async/await in Python." },
    ],
});

const usage = response.usage;
const cached = (usage as any)?.prompt_tokens_details?.cached_tokens ?? 0;
console.log(`Cached tokens: ${cached}`);
console.log(`Total prompt tokens: ${usage?.prompt_tokens}`);

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="claude-sonnet-4-20250514",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant..."},
        {"role": "user", "content": "Explain async/await in Python."}
    ]
)

usage = response.usage
if hasattr(usage, "prompt_tokens_details") and usage.prompt_tokens_details:
    cached = getattr(usage.prompt_tokens_details, "cached_tokens", 0)
    print(f"Cached tokens: {cached}")
print(f"Total prompt tokens: {usage.prompt_tokens}")

import { Client } from "@arbytra/sdk";

const client = new Client({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseUrl: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "claude-sonnet-4-20250514",
    messages: [
        { role: "system", content: "You are a helpful coding assistant..." },
        { role: "user", content: "Explain async/await in Python." },
    ],
});

const cached = response.usage?.prompt_tokens_details?.cached_tokens ?? 0;
console.log(`Cached tokens: ${cached}`);
console.log(`Total prompt tokens: ${response.usage?.prompt_tokens}`);

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "messages": [
      {"role": "system", "content": "You are a helpful coding assistant..."},
      {"role": "user", "content": "Explain async/await in Python."}
    ]
  }'

Override caching per provider

Arbytra handles caching automatically for supported providers. For explicit control, each provider accepts specific fields. When you supply one, Arbytra skips automatic injection and uses your value.

Provider	Field	Effect
Anthropic	`cache_control: {"type": "ephemeral"}` on content blocks	Marks specific content for caching
OpenAI	`prompt_cache_key` (string)	Improves cache hit rate for repeated conversations
OpenAI	`prompt_cache_retention: "24h"`	Extends cache lifetime to 24 hours
Fireworks	`user` (string)	Improves cache reuse across conversation turns

When you provide any of these fields, Arbytra skips automatic cache injection for that provider.

Anthropic — `cache_control`

Add cache_control to content blocks to mark specific content for caching:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1",
)

response = client.chat.completions.create(
    model="claude-sonnet-4-20250514",
    messages=[
        {"role": "system", "content": [
            {"type": "text", "text": "You are a helpful coding assistant with deep knowledge of Python, JavaScript, and Rust. You follow best practices and explain your reasoning step by step.",
             "cache_control": {"type": "ephemeral"}},
        ]},
        {"role": "user", "content": "Explain async/await in Python."},
    ],
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "claude-sonnet-4-20250514",
    messages: [
        { role: "system", content: [
            { type: "text", text: "You are a helpful coding assistant with deep knowledge of Python, JavaScript, and Rust. You follow best practices and explain your reasoning step by step.",
              cache_control: { type: "ephemeral" } },
        ]},
        { role: "user", content: "Explain async/await in Python." },
    ],
});

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="claude-sonnet-4-20250514",
    messages=[
        {"role": "system", "content": [
            {"type": "text", "text": "You are a helpful coding assistant with deep knowledge of Python, JavaScript, and Rust. You follow best practices and explain your reasoning step by step.",
             "cache_control": {"type": "ephemeral"}},
        ]},
        {"role": "user", "content": "Explain async/await in Python."}
    ]
)

import { Client } from "@arbytra/sdk";

const client = new Client({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseUrl: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "claude-sonnet-4-20250514",
    messages: [
        { role: "system", content: [
            { type: "text", text: "You are a helpful coding assistant with deep knowledge of Python, JavaScript, and Rust. You follow best practices and explain your reasoning step by step.",
              cache_control: { type: "ephemeral" } },
        ]},
        { role: "user", content: "Explain async/await in Python." },
    ],
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "messages": [
      {"role": "system", "content": [
        {"type": "text", "text": "You are a helpful coding assistant with deep knowledge of Python, JavaScript, and Rust. You follow best practices and explain your reasoning step by step.",
         "cache_control": {"type": "ephemeral"}}
      ]},
      {"role": "user", "content": "Explain async/await in Python."}
    ]
  }'

The only supported type is "ephemeral". This follows the provider’s default retention behavior. cache_control applies to Anthropic models only. For other providers, automatic optimization handles caching.

OpenAI — `prompt_cache_key` and `prompt_cache_retention`

prompt_cache_key improves cache hit rate for repeated conversations. prompt_cache_retention: "24h" extends the cache lifetime to 24 hours. prompt_cache_retention is supported on gpt-4.1+ and gpt-5+ models only. It isn’t compatible with ZDR data policy. Omit it if your workspace uses ZDR.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1",
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain async/await in Python."},
    ],
    extra_body={
        "prompt_cache_key": "my-conversation-123",
        "prompt_cache_retention": "24h",
    },
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-4.1-mini",
    messages: [
        { role: "system", content: "You are a helpful coding assistant." },
        { role: "user", content: "Explain async/await in Python." },
    ],
    prompt_cache_key: "my-conversation-123",
    prompt_cache_retention: "24h",
});

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain async/await in Python."}
    ],
    prompt_cache_key="my-conversation-123",
    extra_body={"prompt_cache_retention": "24h"}
)

import { Client } from "@arbytra/sdk";

const client = new Client({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseUrl: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-4.1-mini",
    messages: [
        { role: "system", content: "You are a helpful coding assistant." },
        { role: "user", content: "Explain async/await in Python." },
    ],
    prompt_cache_key: "my-conversation-123",
    extra_body: { prompt_cache_retention: "24h" },
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1-mini",
    "messages": [
      {"role": "system", "content": "You are a helpful coding assistant."},
      {"role": "user", "content": "Explain async/await in Python."}
    ],
    "prompt_cache_key": "my-conversation-123",
    "prompt_cache_retention": "24h"
  }'

Fireworks — `user`

On Fireworks, requests with the same user value benefit from improved cache reuse across conversation turns.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1",
)

response = client.chat.completions.create(
    model="gpt-oss-20b",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain async/await in Python."},
    ],
    user="my-conversation-123",
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-oss-20b",
    messages: [
        { role: "system", content: "You are a helpful coding assistant." },
        { role: "user", content: "Explain async/await in Python." },
    ],
    user: "my-conversation-123",
});

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-oss-20b",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain async/await in Python."}
    ],
    user="my-conversation-123"
)

import { Client } from "@arbytra/sdk";

const client = new Client({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseUrl: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-oss-20b",
    messages: [
        { role: "system", content: "You are a helpful coding assistant." },
        { role: "user", content: "Explain async/await in Python." },
    ],
    user: "my-conversation-123",
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-20b",
    "messages": [
      {"role": "system", "content": "You are a helpful coding assistant."},
      {"role": "user", "content": "Explain async/await in Python."}
    ],
    "user": "my-conversation-123"
  }'

Check cache usage

For /v1/chat/completions responses, cache hit information appears in usage.prompt_tokens_details:

{
  "usage": {
    "prompt_tokens": 1500,
    "completion_tokens": 200,
    "total_tokens": 1700,
    "prompt_tokens_details": {
      "cached_tokens": 1200,
      "cache_write_tokens": 300
    }
  }
}

cached_tokens shows how many prompt tokens were served from cache. Arbytra normalizes this field across all providers in the OpenAI-format response. cache_write_tokens indicates how many tokens were written to prompt cache on this request. Present when the provider reports cache-write accounting. For /v1/messages responses, cache tokens appear as top-level usage fields:

{
  "usage": {
    "input_tokens": 300,
    "output_tokens": 200,
    "cache_read_input_tokens": 1200,
    "cache_creation_input_tokens": 0
  }
}

input_tokens represents only the non-cached portion. Total input tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens. For /v1/responses requests, cache token counts appear in usage.input_tokens_details:

{
  "usage": {
    "input_tokens": 1500,
    "output_tokens": 200,
    "total_tokens": 1700,
    "input_tokens_details": {
      "cached_tokens": 1200,
      "cache_write_tokens": 300
    }
  }
}

cache_write_tokens indicates how many tokens were written to prompt cache. Present when the provider reports cache-write accounting.

Check cache savings

Cache savings appear in routing_metadata.cost when savings are greater than zero:

{
  "routing_metadata": {
    "cost": {
      "usd": 0.0042,
      "cache_savings_percent": 47,
      "cache_savings_usd": 0.0037
    }
  }
}

cache_savings_percent is an integer (0-100) showing the percentage saved compared to uncached cost. cache_savings_usd shows the dollar amount saved.

Check cache usage in streams

Cache metrics appear in the final streaming chunk alongside usage and routing_metadata. See Streaming for details on consuming trailing chunks.

Improve cache hits

You can improve cache hit rates by structuring your requests for reuse.

Long, stable system prompts: Place reusable instructions in the system message. The prompt prefix is what providers cache.
Few-shot examples: Static example blocks are reused across requests.
Static before dynamic: Put content that doesn’t change before content that does.
Multi-turn conversations: Shared prompt prefixes get better cache reuse across requests.
Steady request cadence: Providers expire cached tokens after inactivity. Steady flow keeps entries warm.

See Cost optimization for more strategies.

Look up cache pricing

The model directory exposes cache pricing for every supported provider. Query it to see cache_read_price, cache_write_price, and supports_prompt_caching per model:

import os
import httpx

response = httpx.get(
    "https://api.arbytra.com/v1/directory/models",
    headers={"Authorization": f"Bearer {os.environ['ARBYTRA_API_KEY']}"},
)

for model_id, model in response.json()["models"].items():
    for provider in model.get("providers", []):
        for tier in provider.get("tiers", []):
            if tier.get("cache_read_price"):
                print(f"{model_id} ({provider['provider']}): "
                      f"read=${tier['cache_read_price']}/M, "
                      f"write=${tier.get('cache_write_price', 'N/A')}/M")

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

directory = client.models.list_directory()

for model_id, model in directory.models.items():
    for provider in model.providers:
        for tier in provider.tiers:
            if tier.cache_read_price:
                print(f"{model_id} ({provider.provider}): "
                      f"read=${tier.cache_read_price}/M, "
                      f"write=${tier.cache_write_price}/M")

import { Client } from "@arbytra/sdk";

const client = new Client({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseUrl: "https://api.arbytra.com/v1",
});

const directory = await client.models.listDirectory();

for (const [modelId, model] of Object.entries(directory.models)) {
    for (const provider of model.providers) {
        for (const tier of provider.tiers) {
            if (tier.cache_read_price) {
                console.log(`${modelId} (${provider.provider}): ` +
                    `read=$${tier.cache_read_price}/M, ` +
                    `write=$${tier.cache_write_price}/M`);
            }
        }
    }
}

curl https://api.arbytra.com/v1/directory/models \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  | jq '.models | to_entries[] | .value.providers[] | .tiers[] | select(.cache_read_price) | {model: .name, cache_read_price, cache_write_price}'

Providers offer discounted rates for cache reads compared to standard input pricing. Some charge a surcharge for cache writes. Check the directory for current prices.

Troubleshoot

Symptom	Fix
`cached_tokens` always 0 (first request)	The first request creates the cache. Send a follow-up with the same prefix.
`cached_tokens` always 0 (unsupported model)	Check `supports_prompt_caching` in the model directory.
`cached_tokens` always 0 (unique prompts)	Caching requires a shared prefix. Add a reusable system prompt.
`cached_tokens` always 0 (short prompt)	Your prompt may be below the provider’s minimum token threshold. Add more reusable content to the system message.
Lower-than-expected savings	Move static content before dynamic content in messages.
Lower-than-expected savings (gaps between requests)	Providers expire cached tokens after inactivity. Maintain steady request flow.
`cache_savings_percent` not in response	The field appears only when savings are greater than zero.

Resources

Cost optimization — cache economics in routing
Streaming — cache metrics in streaming responses
Model directory — cache pricing and support per model
Response metadata — routing_metadata.cost fields

Get Started

Guides

Response API

Frameworks

Integrations

Specifications

Platform

Changelog

Prerequisites

How it works

Send a cached request

Override caching per provider

Anthropic — `cache_control`

OpenAI — `prompt_cache_key` and `prompt_cache_retention`

Fireworks — `user`

Check cache usage

Check cache savings

Check cache usage in streams

Improve cache hits

Look up cache pricing

Troubleshoot

Resources

​Prerequisites

​How it works

​Send a cached request

​Override caching per provider

​Anthropic — cache_control

​OpenAI — prompt_cache_key and prompt_cache_retention

​Fireworks — user

​Check cache usage

​Check cache savings

​Check cache usage in streams

​Improve cache hits

​Look up cache pricing

​Troubleshoot

​Resources

Prerequisites

How it works

Send a cached request

Override caching per provider

Anthropic — `cache_control`

OpenAI — `prompt_cache_key` and `prompt_cache_retention`

Fireworks — `user`

Check cache usage

Check cache savings

Check cache usage in streams

Improve cache hits

Look up cache pricing

Troubleshoot

Resources