Routing Options

Arbytra selects a provider for each request based on your routing configuration. Pass routing options in the gateway.routing object to control strategy, constraints, and provider filtering.

Prerequisites

An Arbytra API key
Python 3.10+ with the OpenAI SDK (pip install openai) or the arbytra SDK (pip install arbytra)
- OR Node.js 18+ with the OpenAI SDK (npm install openai) or @arbytra/sdk (npm install @arbytra/sdk)

Compare strategies

Arbytra supports seven optimization strategies:

Strategy	Description	Best For
`cost`	Cost-optimized, well-rounded	Cost-conscious production, budget-sensitive apps
`cost-focus` (default)	Aggressively minimize cost	Maximum cost savings, no latency requirements
`ttft`	TTFT-optimized, well-rounded	Streaming UX, interactive apps
`ttft-focus`	Aggressively minimize time to first token	Real-time applications, chatbots
`tps`	Throughput-optimized, well-rounded	High-volume processing
`tps-focus`	Aggressively maximize tokens per second	Maximum throughput, pipeline processing
`balanced`	All dimensions weighted evenly	General-purpose, mixed workloads

Base vs. focus

Base strategies (cost, ttft, tps) optimize for the named dimension while still considering other quality factors. Focus strategies (cost-focus, ttft-focus, tps-focus) optimize almost entirely for the named dimension. Other factors have minimal influence.

Type	Behavior	Use when
Base (`cost`, `ttft`, `tps`)	Favors the named dimension, well-rounded	Production workloads needing reliable performance
Focus (`cost-focus`, `ttft-focus`, `tps-focus`)	Aggressively optimizes the named dimension	Batch processing, real-time streaming UI, high-throughput pipelines

For custom weight configurations beyond the preset strategies, see Set custom weights.

Optimize for cost

Arbytra computes the expected cost of each request at every available provider and routes to the cheapest one. The cost model accounts for caching and pricing tiers. See Cost optimization for configuration, code examples, and the full cost model.

Optimize for latency

Route requests to low-latency providers:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Quick answer: 2+2?"}],
    extra_body={"gateway": {"routing": {"optimize": "ttft-focus"}}}
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Quick answer: 2+2?" }],
    gateway: { routing: { optimize: "ttft-focus" } },
});

console.log(response.choices[0].message.content);

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Quick answer: 2+2?"}],
    gateway={
        "routing": {
            "optimize": "ttft-focus",
        },
    }
)

print(f"Provider: {response.routing_metadata.provider}")

const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Quick answer: 2+2?" }],
    gateway: {
        routing: {
            optimize: "ttft-focus",
        },
    },
});

console.log(`Provider: ${response.routing_metadata?.provider}`);

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Quick answer: 2+2?"}],
    "gateway": {"routing": {"optimize": "ttft-focus"}}
  }'

Set latency constraints

Set maximum time-to-first-token (TTFT):

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {
        "optimize": "cost",
        "max_ttft_ms": 1000,
        "ttft_percentile": "p50",
    }}}
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: {
        optimize: "cost",
        max_ttft_ms: 1000,
        ttft_percentile: "p50",
    } },
});

console.log(response.choices[0].message.content);

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "optimize": "cost",
            "max_ttft_ms": 1000,  # Must start responding within 1s
            "ttft_percentile": "p50",
        },
    }
)

const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            optimize: "cost",
            max_ttft_ms: 1000, // Must start responding within 1s
            ttft_percentile: "p50",
        },
    },
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"optimize": "cost", "max_ttft_ms": 1000, "ttft_percentile": "p50"}}
  }'

If no provider can meet the latency constraint, Arbytra returns a 400 error.

max_ttft_ms evaluates against median (p50) metrics by default. To constrain on worst-case latency, set ttft_percentile to "p95". See Choose metric percentile.

Set cost ceilings

Exclude providers that exceed a per-1M-token budget:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {
        "optimize": "cost",
        "max_cost_per_1m": 10.00,
    }}}
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: {
        optimize: "cost",
        max_cost_per_1m: 10.00,
    } },
});

console.log(response.choices[0].message.content);

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "optimize": "cost",
            "max_cost_per_1m": 10.00,  # Max $10 per 1M tokens (average of input + output)
        },
    }
)

const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            optimize: "cost",
            max_cost_per_1m: 10.0, // Max $10 per 1M tokens (average of input + output)
        },
    },
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"optimize": "cost", "max_cost_per_1m": 10.00}}
  }'

Arbytra calculates cost as the average of input and output price per 1M tokens. Providers exceeding this ceiling are excluded from routing. For fine-grained constraints, see Advanced routing and Cost optimization.

Require supported parameters

Set require_parameters to true to only route to providers that accept all optional parameters you sent (like seed, logit_bias, or top_logprobs). Without this flag, Arbytra drops unsupported parameters and adds a warning to the response.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    seed=42,
    extra_body={"gateway": {"routing": {
        "optimize": "cost",
        "require_parameters": True,
    }}}
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello!" }],
    seed: 42,
    gateway: { routing: {
        optimize: "cost",
        require_parameters: true,
    } },
});

console.log(response.choices[0].message.content);

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    seed=42,
    gateway={
        "routing": {
            "optimize": "cost",
            "require_parameters": True,
        },
    }
)

const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello!" }],
    seed: 42,
    gateway: {
        routing: {
            optimize: "cost",
            require_parameters: true,
        },
    },
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "seed": 42,
    "gateway": {"routing": {"optimize": "cost", "require_parameters": true}}
  }'

If no provider supports the parameters you sent, Arbytra returns a 400 error with code required_params_not_supported. See Filter by parameter support for the full list of parameters this applies to.

Prefer or exclude providers

Prefer or exclude specific providers:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

# Only consider these providers
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {"providers": ["openai", "anthropic"]}}}
)

# Exclude providers
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {"exclude_providers": ["deepseek"]}}}
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

// Only consider these providers
const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: { providers: ["openai", "anthropic"] } },
});

// Exclude providers
const response2 = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: { exclude_providers: ["deepseek"] } },
});

console.log(response.choices[0].message.content);

# Only consider these providers
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "providers": ["openai", "anthropic"],
        },
    }
)

# Exclude providers
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "exclude_providers": ["deepseek"],
        },
    }
)

// Only consider these providers
const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            providers: ["openai", "anthropic"],
        },
    },
});

// Exclude providers
const response2 = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            exclude_providers: ["deepseek"],
        },
    },
});

# Only consider these providers
curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"providers": ["openai", "anthropic"]}}
  }'

# Exclude providers
curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"exclude_providers": ["deepseek"]}}
  }'

You can hint at a preferred provider without restricting the candidate pool:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {"prefer": "openai"}}}
)

const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: { prefer: "openai" } },
});

console.log(response.choices[0].message.content);

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "prefer": "openai",
        },
    }
)

const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            prefer: "openai",
        },
    },
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"prefer": "openai"}}
  }'

prefer is a soft hint. If the preferred provider is available, Arbytra routes to it. If not, routing proceeds normally. Unlike providers, a prefer miss doesn’t fail the request.

Restrict key source

Force requests to use only BYOK (bring-your-own-key) or only platform-managed keys:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

# Use only your own provider keys
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {"only_byok": True}}}
)

# Use only Arbytra platform keys
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {"only_platform": True}}}
)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

// Use only your own provider keys
const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: { only_byok: true } },
});

// Use only Arbytra platform keys
const response2 = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: { only_platform: true } },
});

console.log(response.choices[0].message.content);

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

# Use only your own provider keys
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "only_byok": True,
        },
    }
)

# Use only Arbytra platform keys
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "only_platform": True,
        },
    }
)

// Use only your own provider keys
const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            only_byok: true,
        },
    },
});

// Use only Arbytra platform keys
const response2 = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            only_platform: true,
        },
    },
});

# Use only your own provider keys
curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"only_byok": true}}
  }'

# Use only Arbytra platform keys
curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"only_platform": true}}
  }'

Both are booleans, default false. Setting both to true returns a 400 error. They’re mutually exclusive. When no key of the requested type is available, the request fails with no fallback. See Bring Your Own Key for BYOK setup.

Opt in to premium tiers

Premium-tier offerings are excluded from routing by default to prevent accidental cost escalation. Set tier to opt in.

Value	Effect
`"priority"`	Includes Anthropic Fast Mode offerings (2.5x speed, 6x cost)
omitted (default)	Excludes premium-tier offerings

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="claude-opus-4-6",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"gateway": {"routing": {"tier": "priority"}}}
)

const response = await client.chat.completions.create({
    model: "claude-opus-4-6",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: { routing: { tier: "priority" } },
});

console.log(response.choices[0].message.content);

response = client.chat.completions.create(
    model="claude-opus-4-6",
    messages=[{"role": "user", "content": "Hello!"}],
    gateway={
        "routing": {
            "tier": "priority",
        },
    }
)

const response = await client.chat.completions.create({
    model: "claude-opus-4-6",
    messages: [{ role: "user", content: "Hello!" }],
    gateway: {
        routing: {
            tier: "priority",
        },
    },
});

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-6",
    "messages": [{"role": "user", "content": "Hello!"}],
    "gateway": {"routing": {"tier": "priority"}}
  }'

Arbytra’s “priority” tier refers to Anthropic Fast Mode, not Anthropic’s separate Priority Tier (committed capacity SLA).

Without tier, requests to models available only under a premium tier return tier_opt_in_required.

Read routing metadata

Every response carries routing information:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

metadata = response.routing_metadata
print(f"Provider: {metadata.provider}")
print(f"Model: {metadata.provider_model_id}")
print(f"Strategy: {metadata.routing_strategy}")
print(f"Cost: ${metadata.cost.usd:.6f}")

const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello!" }],
});

const metadata = response.routing_metadata;
console.log(`Provider: ${metadata?.provider}`);
console.log(`Model: ${metadata?.provider_model_id}`);
console.log(`Strategy: ${metadata?.routing_strategy}`);
console.log(`Cost: $${metadata?.cost?.usd}`);

For routing metadata with the OpenAI SDK, see OpenAI Compatibility. For the complete field reference including fallback chain, warnings, and all optional fields, see Response Extensions.

Combine routing options

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What's the capital of France?"}
    ],
    extra_body={"gateway": {"routing": {
        "optimize": "ttft-focus",
        "max_ttft_ms": 1000,
    }}}
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseURL: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "What's the capital of France?" },
    ],
    gateway: { routing: {
        optimize: "ttft-focus",
        max_ttft_ms: 1000,
    } },
});

console.log(response.choices[0].message.content);

import os
from arbytra import Client

client = Client(
    api_key=os.environ["ARBYTRA_API_KEY"],
    base_url="https://api.arbytra.com/v1"
)

# For a chatbot: optimize TTFT with cost ceiling
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What's the capital of France?"}
    ],
    gateway={
        "routing": {
            "optimize": "ttft-focus",
            "max_ttft_ms": 1000,
        },
    }
)

print(response.choices[0].message.content)
print(f"\n--- Routing Info ---")
print(f"Provider: {response.routing_metadata.provider}")
print(f"Strategy: {response.routing_metadata.routing_strategy}")
print(f"Cost: ${response.routing_metadata.cost.usd:.6f}")

import { Client } from "@arbytra/sdk";

const client = new Client({
    apiKey: process.env.ARBYTRA_API_KEY,
    baseUrl: "https://api.arbytra.com/v1",
});

const response = await client.chat.completions.create({
    model: "gpt-5.4",
    messages: [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: "What's the capital of France?" },
    ],
    gateway: {
        routing: {
            optimize: "ttft-focus",
            max_ttft_ms: 1000,
        },
    },
});

console.log(response.choices[0].message.content);
console.log(`\n--- Routing Info ---`);
console.log(`Provider: ${response.routing_metadata?.provider}`);
console.log(`Strategy: ${response.routing_metadata?.routing_strategy}`);
console.log(`Cost: $${response.routing_metadata?.cost?.usd}`);

curl https://api.arbytra.com/v1/chat/completions \
  -H "Authorization: Bearer $ARBYTRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What'\''s the capital of France?"}
    ],
    "gateway": {"routing": {"optimize": "ttft-focus", "max_ttft_ms": 1000}}
  }'

Get Started

Guides

Response API

Frameworks

Integrations

Specifications

Platform

Changelog

Routing Options

Prerequisites

Compare strategies

Base vs. focus

Optimize for cost

Optimize for latency

Set latency constraints

Set cost ceilings

Require supported parameters

Prefer or exclude providers

Restrict key source

Opt in to premium tiers

Read routing metadata

Combine routing options

​Prerequisites

​Compare strategies

​Base vs. focus

​Optimize for cost

​Optimize for latency

​Set latency constraints

​Set cost ceilings

​Require supported parameters

​Prefer or exclude providers

​Restrict key source

​Opt in to premium tiers

​Read routing metadata

​Combine routing options

Prerequisites

Compare strategies

Base vs. focus

Optimize for cost

Optimize for latency

Set latency constraints

Set cost ceilings

Require supported parameters

Prefer or exclude providers

Restrict key source

Opt in to premium tiers

Read routing metadata

Combine routing options