Production practices

A successful development request is only the beginning. Production integration also requires credential isolation, timeouts, retries, observability, capacity control, and graceful degradation.

Isolate environments

Use separate API keys for development, staging, and production.

Do not share quota limits and routing policies across unrelated environments.

Configure instead of hard-coding

Keep these values configurable:

• Base URL
• API key
• model ID
• timeout
• maximum retries
• concurrency limits
• fallback model or degradation policy

Timeouts

Do not treat every timeout as one number. Where possible, distinguish connection time, time to first byte, streaming lifetime, and overall task duration.

Text chat and long reasoning or media generation need different budgets.

Retries

Retry only appropriate failures, such as selected 429 and transient 5xx responses.

Avoid unlimited retries, rapid replay loops, retrying every 4xx, or blindly replaying expensive generation requests after a network interruption.

Use exponential backoff with jitter.

Concurrency and backpressure

Protect your own service with concurrency limits, queues, user-level rate limits, cancellation, and circuit-breaker or degradation logic where appropriate.

Graceful degradation

Critical workloads should not depend on a single hard-coded model.

Primary model
    ↓ unavailable
Fallback model
    ↓ unavailable
Graceful response / queue / manual review

Validate fallback quality, protocol compatibility, and cost before production.

Observability

Useful fields include request time, business request ID, model, endpoint, HTTP status, latency, retry count, and cost-related summaries where appropriate.

Never log full API keys. Whether prompts and responses are logged should follow your own privacy and compliance requirements.

Key rotation

A safer rotation process is:

1. create a new key,
2. deploy it alongside the old key,
3. verify traffic,
4. revoke the old key,
5. confirm the old key no longer receives requests.

Pre-production checklist

• [ ] production and development keys are separate
• [ ] model and Base URL are configurable
• [ ] 429 / 5xx retries are bounded
• [ ] requests have appropriate timeouts
• [ ] logs do not expose secrets
• [ ] concurrency is controlled
• [ ] a fallback or degradation path exists
• [ ] actual endpoint support and pricing have been verified