Cloudflare introduces its AI-focused product suite: Check features
What's the story
Cloudflare, a top cloud services provider, has launched a collection of AI-focused products designed to make it easier to create, deploy, and operate AI models.
The new line-up includes Workers AI, which lets users access nearby GPUs hosted by Cloudflare partners on a pay-as-you-go basis.
Another product called AI Gateway, offers 'observability features' to understand AI traffic like the number of requests and, cost of running the app, among others.
Details
Workers AI offers low-latency AI-powered user experience
Workers AI is crafted to make sure AI inference takes place on GPUs that are geographically close to users.
It uses ONNX, a machine learning toolkit backed by Microsoft, to enable AI models to run wherever processing is most efficient in terms of bandwidth, latency, connectivity, and localization constraints.
Users can select from a range of models, including large language models like Meta's Llama 2, automatic speech recognition models, image classifiers, and sentiment analysis models.
Insights
AI Gateway ensures cost management
AI Gateway provides features for tracking AI traffic, including cost of running an AI app.
It also offers capabilities to cut costs through caching and rate limiting. With caching, customers can cache responses from large language models to common questions, reducing the need for generating new responses.
Rate limiting allows for better control over app scaling by mitigating malicious actors and heavy traffic.
What Next?
Vectorize stores vector embeddings for AI models
Another product, called Vectorize is designed for customers who need to store vector embeddings for their AI models in a database.
These embeddings are compact representations of training data that preserve the meaningful aspects of the data and are used by various applications such as search and AI assistants.
Models in Workers AI can create embeddings that can then be stored in Vectorize. Plus, customers can store embeddings created by third-party models from vendors like OpenAI and Cohere.
Official words
Current offerings in the market are complicated: Cloudflare CEO
"The offerings already on the market are still very complicated — they require stitching together lots of new vendors, and it gets expensive fast," Cloudflare CEO Matthew Prince told TechCrunch.
"There's also very little insight currently available on how you're spending money on AI; observability is a big challenge as AI spend skyrockets. We can help simplify all of these aspects for developers, " he added.