Claude API 500 api_error and 502 Bad Gateway — a retry policy that actually recovers

500 and 502 mean a fault, not a limit. That makes them the one error class in this family worth retrying almost immediately — unlike 429, which is your own rate limit, and 529, which is the provider being full. Retrying the wrong one of the three turns a small incident into your incident.

The error

two shapes, same remedy

// Straight from the model provider (HTTP 500)
{
  "type": "error",
  "error": { "type": "api_error", "message": "Internal server error" }
}

// From a gateway or proxy in between (HTTP 502)
{
  "error": {
    "message": "Failed to reach upstream provider",
    "type": "upstream_error",
    "code": "upstream_error",
    "param": null
  }
}

Causes and fixes at a glance

Cause	Fix
Transient provider-side fault	Retry with exponential backoff and jitter, capped at ~5 attempts.
Connection accepted, then never answered	A hang, not an error. Bound time-to-first-byte separately from total duration.
An intermediary returning its own 502	Nothing to do with the model. Check whether the body is the provider's shape or a proxy's.
Retrying blindly during a real incident	Cap attempts and back off — otherwise your retries become part of the outage.

Separate 500 from 529 and 429 before choosing a remedy

429 is a rate limit you are exceeding — slow down. 529 is the provider at capacity — back off much harder and longer. 500/502 is a fault, usually brief and often specific to one request. Only the third is worth retrying quickly, and treating all three the same is why retry loops make incidents worse.

Retry 5xx, never 4xx

Exponential backoff with jitter, five attempts maximum. The same helper works for every provider — a 400 or 422 will fail identically on the next attempt, so retrying it only spends latency to reach the same error.

backoff.py

import time, random
from openai import OpenAI, APIStatusError

client = OpenAI(base_url="https://api.kunavo.com/v1", api_key="sk-kn-...")

def with_backoff(fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return fn()
        except APIStatusError as e:
            if e.status_code not in (429, 500, 529):
                raise                      # don't retry auth/validation errors
            retry_after = e.response.headers.get("retry-after")
            delay = float(retry_after) if retry_after else min(2 ** attempt, 30)
            time.sleep(delay + random.uniform(0, 0.5))   # jitter avoids herds
    raise RuntimeError("retries exhausted")

resp = with_backoff(lambda: client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "ping"}],
    max_tokens=32,
))
print(resp.choices[0].message.content)

Bound time-to-first-byte separately from total duration

One timeout for the whole request cannot distinguish a long generation from a dead connection. Set a short deadline for the first byte and a generous one for the rest; a hang then fails fast while a genuinely slow answer is left alone.

Log status and latency per attempt

Without per-attempt records, a provider incident and your own timeout look identical after the fact. Status code, latency and attempt number are enough to tell them apart the next morning.

If you’re calling through Kunavo

On Kunavo a 5xx from the channel serving your request is retried inside the same request, on that model's other configured channel, before any error reaches you — on /v1/messages, /v1/responses, and Claude models on /v1/chat/completions, for models that have a second channel. Two honest limits: the retry is decided from response headers alone, so it can only happen before the first streamed byte — once a stream has started, a mid-stream failure is yours to handle; and models with only one channel configured are attempted once. When both channels fail you get the first one's error, and the request is recorded at zero cost. The routing behind that behaviour is described in our AI gateway guide.

FAQ

Do I get billed for a request that returns 500?

On Kunavo, no — failed requests are recorded at zero cost. Billing direct with a provider varies, but a 5xx generally is not charged.

Can retrying a 500 produce two completions?

Yes. A request can fail after the model has already generated. If the work has side effects, make it idempotent at your layer before adding retries.

What is the one-line difference between 500, 502 and 529?

500 is the provider faulting, 502 is something in front of it failing to reach the provider, and 529 is the provider being at capacity — retry the first two soon, the third much later.

Related guides

More error semantics live in the error reference; getting a key takes a minute via sign up and the authentication docs.