Cloudflare’s AI Platform: an inference layer designed for agents

Summary

Cloudflare is expanding its AI platform into a unified inference layer, providing a single API to access models from various third-party providers. This update enables developers to manage multi-model workflows, centralize cost monitoring, and deploy custom containerized models through a consistent interface.

Key Points

• Provides a unified API via the AI.run() binding to access 70+ models from over 12 providers, including OpenAI, Anthropic, Google, and Alibaba Cloud.
• Supports multimodal applications through the inclusion of image, video, and speech models in the catalog.
• Enables centralized cost management via AI Gateway, allowing for granular tracking using custom metadata such as teamId or userId.
• Introduces support for custom model deployment to Workers AI using Replicate’s Cog technology for containerization.
• Implements automatic failover and retries via AI Gateway to mitigate upstream provider outages.
• Features streaming response buffering in AI Gateway to maintain session continuity for long-running agents using the Agents SDK.

Technical Details

The platform allows developers to switch between Cloudflare-hosted models and third-party models (e.g., anthropic/claude-opus-4-6) by modifying the gateway object within the AI.run() call. While currently accessible via Workers bindings, a REST API for accessing the model catalog from any environment is forthcoming.

For custom model deployment, the platform utilizes Cog, a tool that abstracts machine learning packaging. Developers define dependencies in a cog.yaml file (specifying Python versions and requirements) and implement inference logic in a predict.py file. After running cog build to create a container image, the image can be pushed to Workers AI for managed deployment.

To optimize latency, Cloudflare leverages its global network of data centers in 330 cities to reduce the "time to first token." For agentic workflows, AI Gateway buffers streaming responses independently of the agent's lifecycle. This ensures that if an agent is interrupted, it can reconnect and retrieve the buffered response without triggering a new, costly inference call.

Impact / Why It Matters

This unified layer reduces vendor lock-in and operational complexity by allowing developers to orchestrate multiple models through a single endpoint. It also simplifies the scaling of specialized AI applications by providing a standardized path for deploying fine-tuned, containerized models.