Multi-Provider Routing

Build flexible AI applications that work with multiple providers by abstracting provider logic, implementing fallback strategies, and routing requests based on cost, speed, or feature requirements.

Multi-provider architecture provides resilience. By supporting multiple AI providers, your application can gracefully handle rate limits, outages, and pricing changes without code changes.

For automatic failover, use built-in fallbacks. The rewrite endpoint supports automatic provider failover via the x-lava-fallbacks header — no try/catch required. The patterns on this page are for cases where you need custom routing logic beyond simple failover.

Provider Configuration Pattern

Create a configuration-driven approach instead of hardcoding providers:

// config/providers.ts
export interface ProviderConfig {
  name: string;
  baseUrl: string;
  models: string[];
  features: string[];
  priority: number;
}

export const PROVIDERS: Record<string, ProviderConfig> = {
  openai: {
    name: 'OpenAI',
    baseUrl: 'https://api.openai.com/v1/chat/completions',
    models: ['gpt-4o', 'gpt-4o-mini', 'o3-mini'],
    features: ['streaming', 'function-calling', 'vision'],
    priority: 1
  },
  anthropic: {
    name: 'Anthropic',
    baseUrl: 'https://api.anthropic.com/v1/messages',
    models: ['claude-opus-4-5-20250514', 'claude-sonnet-4-5-20250514', 'claude-haiku-3-5-20241022'],
    features: ['streaming', 'function-calling', 'vision'],
    priority: 2
  },
  google: {
    name: 'Google',
    baseUrl: 'https://generativelanguage.googleapis.com/v1beta/models',
    models: ['gemini-2.5-flash', 'gemini-2.0-flash'],
    features: ['streaming', 'vision', 'multimodal'],
    priority: 3
  },
  groq: {
    name: 'Groq',
    baseUrl: 'https://api.groq.com/openai/v1/chat/completions',
    models: ['llama-3.3-70b-versatile', 'llama-3.1-8b-instant'],
    features: ['streaming', 'low-latency'],
    priority: 4
  }
};

For a complete provider abstraction class that wraps this configuration with unified request/response handling, see the full example in the Node.js SDK reference.

Provider Switching with Configuration

Environment-Based Selection

Switch providers via environment variables:

// lib/get-provider.ts
import { PROVIDERS } from '@/config/providers';

export function getProviderConfig(providerName?: string) {
  const name = providerName || process.env.AI_PROVIDER || 'openai';
  const config = PROVIDERS[name];

  if (!config) {
    throw new Error(`Unknown provider: ${name}`);
  }

  return config;
}

Environment configuration:

# .env.production
AI_PROVIDER=openai

# .env.development
AI_PROVIDER=groq  # Use faster Groq for development

Dynamic Provider Selection

Choose provider based on request characteristics:

export function selectProvider(request: {
  requiresVision?: boolean;
  requiresFunctionCalling?: boolean;
  prioritizeSpeed?: boolean;
  prioritizeCost?: boolean;
}): string {
  // Vision required
  if (request.requiresVision) {
    return 'google';  // Gemini Pro Vision
  }

  // Speed priority (low latency)
  if (request.prioritizeSpeed) {
    return 'groq';  // Ultra-fast inference
  }

  // Cost priority
  if (request.prioritizeCost) {
    return 'groq';  // Most cost-effective
  }

  // Default: OpenAI for quality
  return 'openai';
}

// Usage
const providerName = selectProvider({ prioritizeSpeed: true });
const config = PROVIDERS[providerName];

Model-Based Routing

Route to providers based on desired model:

export function getProviderForModel(modelName: string): string {
  for (const [providerKey, config] of Object.entries(PROVIDERS)) {
    if (config.models.includes(modelName)) {
      return providerKey;
    }
  }
  throw new Error(`No provider found for model: ${modelName}`);
}

// Usage
const providerName = getProviderForModel('claude-opus-4-5-20250514');
const config = PROVIDERS[providerName];

Fallback Strategies

Sequential Fallback

Try providers in priority order until one succeeds:

export async function completionWithFallback(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  model?: string
): Promise<any> {
  // Sort providers by priority
  const sortedProviders = Object.entries(PROVIDERS)
    .sort(([, a], [, b]) => a.priority - b.priority)
    .map(([key]) => key);

  const errors: Array<{ provider: string; error: string }> = [];

  for (const providerName of sortedProviders) {
    try {
      console.log(`Attempting provider: ${providerName}`);
      const config = PROVIDERS[providerName];

      const response = await fetch(
        `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${forwardToken}`,
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({
            model: model || config.models[0],
            messages,
            temperature: 0.7
          })
        }
      );

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      console.log(`Success with provider: ${providerName}`);
      return await response.json();
    } catch (error) {
      const errorMessage = error instanceof Error ? error.message : 'Unknown error';
      errors.push({ provider: providerName, error: errorMessage });
      console.warn(`Provider ${providerName} failed:`, errorMessage);
      continue;
    }
  }

  // All providers failed
  throw new Error(
    `All providers failed:\n${errors.map(e => `- ${e.provider}: ${e.error}`).join('\n')}`
  );
}

// Usage
try {
  const response = await completionWithFallback(
    forwardToken,
    [{ role: 'user', content: 'Hello' }]
  );
  console.log('Response:', response);
} catch (error) {
  console.error('All providers exhausted:', error);
}

Smart Retry with Exponential Backoff

Add exponential backoff for transient errors before falling back to the next provider:

async function completionWithRetry(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  maxRetries = 3
): Promise<any> {
  const sortedProviders = Object.keys(PROVIDERS)
    .sort((a, b) => PROVIDERS[a].priority - PROVIDERS[b].priority);

  for (const providerName of sortedProviders) {
    let retries = 0;

    while (retries < maxRetries) {
      try {
        const config = PROVIDERS[providerName];
        const response = await fetch(
          `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
          {
            method: 'POST',
            headers: {
              'Authorization': `Bearer ${forwardToken}`,
              'Content-Type': 'application/json'
            },
            body: JSON.stringify({
              model: config.models[0],
              messages
            })
          }
        );

        if (!response.ok) {
          throw new Error(`HTTP ${response.status}`);
        }

        return await response.json();
      } catch (error) {
        const errorMessage = error instanceof Error ? error.message : '';

        // Check if error is retryable (rate limit, timeout)
        const isRetryable = errorMessage.includes('429') ||
                           errorMessage.includes('timeout') ||
                           errorMessage.includes('503');

        if (!isRetryable || retries >= maxRetries - 1) {
          // Non-retryable error or max retries reached, try next provider
          break;
        }

        // Exponential backoff: 1s, 2s, 4s
        const delay = Math.pow(2, retries) * 1000;
        console.log(`Retrying ${providerName} in ${delay}ms...`);
        await new Promise(resolve => setTimeout(resolve, delay));

        retries++;
      }
    }
  }

  throw new Error('All providers and retries exhausted');
}

Cost-Aware Routing

Fallback to cheaper providers when budget is a concern:

interface ProviderWithCost extends ProviderConfig {
  estimatedCostPer1kTokens: number;
}

async function costAwareFallback(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  maxCostUsd: number
): Promise<any> {
  // Try cheapest providers first
  const providersByCost = Object.entries(PROVIDERS)
    .sort(([, a], [, b]) =>
      (a as ProviderWithCost).estimatedCostPer1kTokens -
      (b as ProviderWithCost).estimatedCostPer1kTokens
    );

  for (const [providerName, config] of providersByCost) {
    try {
      const response = await fetch(
        `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${forwardToken}`,
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({
            model: config.models[0],
            messages
          })
        }
      );

      if (!response.ok) throw new Error(`HTTP ${response.status}`);
      return await response.json();
    } catch (error) {
      continue;
    }
  }

  throw new Error(`No provider within budget: $${maxCostUsd}`);
}

Feature Detection

Check provider capabilities before using advanced features:

function hasFeature(providerName: string, feature: string): boolean {
  const config = PROVIDERS[providerName];
  return config?.features.includes(feature) || false;
}

// Usage
if (hasFeature('openai', 'function-calling')) {
  // Include function definitions in request
}

if (hasFeature('google', 'vision')) {
  // Include image URLs in request
}

Graceful Feature Degradation

Fallback when features aren’t supported:

async function completionWithFeatureFallback(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  options: { functions?: any[] } = {}
): Promise<any> {
  const preferredProvider = 'openai';

  try {
    if (options.functions && !hasFeature(preferredProvider, 'function-calling')) {
      console.warn('Function calling not supported, removing from request');
      delete options.functions;
    }

    const config = PROVIDERS[preferredProvider];
    const response = await fetch(
      `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
      {
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${forwardToken}`,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({
          model: config.models[0],
          messages,
          ...(options.functions ? { tools: options.functions } : {})
        })
      }
    );

    if (!response.ok) throw new Error(`HTTP ${response.status}`);
    return await response.json();
  } catch (error) {
    // Fallback to another provider without advanced features
    const fallbackConfig = PROVIDERS['groq'];
    const response = await fetch(
      `https://api.lava.so/v1/forward?u=${encodeURIComponent(fallbackConfig.baseUrl)}`,
      {
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${forwardToken}`,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({
          model: fallbackConfig.models[0],
          messages
        })
      }
    );

    if (!response.ok) throw new Error(`HTTP ${response.status}`);
    return await response.json();
  }
}

Troubleshooting

Fallback doesn't trigger when primary fails

Check:

Exception is being caught properly in try/catch
Provider priority order is correct (lower number = higher priority)
Error is thrown (not just logged) when provider fails

Debug:

for (const providerName of sortedProviders) {
  try {
    console.log('Trying provider:', providerName);
    const response = await makeRequest(providerName);
    console.log('Success!');
    return response;
  } catch (error) {
    console.error('Provider failed:', providerName, error);
    // IMPORTANT: Must continue, not return/throw here
    continue;
  }
}

Feature detection returns wrong capabilities

Issue: Provider claims to support feature but request failsReasons:

Provider configuration outdated (features changed)
Feature available but different API format required
Feature requires specific model (not all models support all features)

Solution: Add model-specific feature checks:

function hasFeature(providerName: string, feature: string, model?: string): boolean {
  const config = PROVIDERS[providerName];
  if (!config?.features.includes(feature)) {
    return false;
  }

  // Model-specific feature support
  if (feature === 'vision' && model) {
    return model.includes('vision') || model.includes('4');
  }

  return true;
}

Cost estimates inaccurate across providers

Problem: Estimated costs don’t match actual Lava costsExplanation:

Lava costs include provider base cost + merchant fee + service charge
Provider pricing varies by model and usage
Costs are only accurate AFTER request completes

Solution: Track usage from response body and calculate cost based on your pricing:

const usage = data.usage?.total_tokens || 0;
const requestId = response.headers.get('x-lava-request-id');

// Calculate cost based on your configured pricing
console.log('Tokens used:', usage, 'Request ID:', requestId);

Response parsing fails for a new provider

Cause: Provider uses different response format than expectedSolution: Handle multiple response formats:

function extractContent(data: any): string {
  // Try each known format
  if (data.choices?.[0]?.message?.content) return data.choices[0].message.content;
  if (data.content?.[0]?.text) return data.content[0].text;
  if (data.candidates?.[0]?.content?.parts?.[0]?.text) {
    return data.candidates[0].content.parts[0].text;
  }

  // Log unknown format for debugging
  console.error('Unknown response format:', JSON.stringify(data, null, 2));
  throw new Error('Unknown response format');
}

Next Steps

Streaming

Handle streaming responses across different providers

Forward Proxy & Authentication

Understand URL encoding and provider routing

Supported Providers

See all supported providers and their features

Webhooks

Receive real-time notifications for usage events

Get Started

AI Gateway

AI Spend

Monetize

Integration

SDK

Cookbook

Comparisons

Provider Configuration Pattern

Provider Switching with Configuration

Environment-Based Selection

Dynamic Provider Selection

Model-Based Routing

Fallback Strategies

Sequential Fallback

Smart Retry with Exponential Backoff

Cost-Aware Routing

Feature Detection

Graceful Feature Degradation

Troubleshooting

Next Steps

Streaming

Forward Proxy & Authentication

Supported Providers

Webhooks

Get Started

AI Gateway

AI Spend

Monetize

Integration

SDK

Cookbook

Comparisons

Documentation Index

​Provider Configuration Pattern

​Provider Switching with Configuration

​Environment-Based Selection

​Dynamic Provider Selection

​Model-Based Routing

​Fallback Strategies

​Sequential Fallback

​Smart Retry with Exponential Backoff

​Cost-Aware Routing

​Feature Detection

​Graceful Feature Degradation

​Troubleshooting

​Next Steps

Streaming

Forward Proxy & Authentication

Supported Providers

Webhooks

Provider Configuration Pattern

Provider Switching with Configuration

Environment-Based Selection

Dynamic Provider Selection

Model-Based Routing

Fallback Strategies

Sequential Fallback

Smart Retry with Exponential Backoff

Cost-Aware Routing

Feature Detection

Graceful Feature Degradation

Troubleshooting

Next Steps