3 The Model Service: Your platform’s gateway to AI models

The Model Service contract groups operations into 5 categories: inference for generating responses, discovery for querying available models, prompt management for storing and retrieving system prompts, a registry for custom models, and embeddings for turning text into vectors.

The Model Service provides a unified interface to any provider. Applications use the service without knowing which provider handles each request. Users configure their own fallback chain and routing logic. This architecture delivers flexibility without opinions. The platform provided adapters, configuration tools and retry mechanisms. Users decide which providers to use and how to handle failure.

Provider adapter class hierarchy. These adapters demonstrate why abstraction matters. OpenAI includes system messages in arrays. Anthropic requires them separately. OpenAI returns content directly. Anthropic returns content blocks. The adapters hide these differences completely.

Cost-aware routing decision flow. The application checks its current spending against the monthly budget first. When near the limit, it switches to free local models. Otherwise, it selects between cheap and powerful models based on task complexity. This ensures the application never exceeds its budget while using the best model it can afford.

Load-based routing flow. The application tracks active requests per provider and routes each new request to the provider with the least load. Counters are incremented before the request and decremented after completion (whether successful or failed), ensuring accurate load tracking.

Feature-based routing flow. The application checks each required feature and identifies which providers support it. The intersection of these provider sets yields the capable providers. If no provider supports all required features, the request fails fast rather than sending it to an incapable provider.

The two levels of caching in LLM applications. Response caching (application level) provides the fastest, cheapest responses by avoiding API calls entirely. Prompt caching (provider level) reduces costs when the same prompt is reused but still requires API calls. Both strategies work together to optimize cost and performance.

Request and response flow from developer code to LLM provider. Developers work with Python objects at the top layer. The SDK's ModelClient converts these to Protocol Buffer messages for gRPC transport. The API Gateway routes requests to the Model Service, which communicates with LLM providers over HTTP. Responses flow back through the same path, with the SDK converting Protocol Buffers back to Python objects.

• The Model Service provides vendor independence through a unified interface that works with any LLM provider, which allows developers to switch between OpenAI, Anthropic, Google, and self-hosted models without changing application code.
• The gRPC contract defines operations covering chat completion, streaming, model discovery, prompt management, custom model registration, and embeddings, establishing the official specification that the SDK, API Gateway, and other services build upon.
• The provider adapter pattern makes providers completely interchangeable from the application's perspective by encapsulating all provider-specific complexity, including message formats, API structures, response parsing, and error handling..
• OpenAI's message format serves as the platform standard because it has become a de facto industry convention, minimizing translation for most providers while giving developers a familiar structure that the Session Service also uses for storage.
• Commercial API adapters for OpenAI, Anthropic, and Google translate between the unified interface and each provider's specific conventions to handle differences like Anthropic's separate system prompt parameter and Google's parts-based message structure.
• Self-hosted model support through vLLM, TGI, Ollama, and other frameworks leverages OpenAI API compatibility, enabling local deployment with the same integration patterns as commercial APIs and requiring only endpoint registration through RegisterModel.
• Structured output through the response_format parameter constrains model responses to specific schemas, which is essential for programmatic parsing, downstream system integration, and classification tasks that feed into routing or policy decisions.
• Streaming improves user experience by delivering tokens as they're generated and enables early termination when responses head in the wrong direction.
• The streaming pipeline flows from provider through adapter, Model Service, gRPC stream, Gateway, and SSE to client, with each layer forwarding chunks immediately rather than buffering, and adapters normalizing the different streaming formats each provider uses.
• Retry configuration controls how the Model Service handles transient failures, specifying maximum attempts, delay timing, exponential backoff, and which error types deserve retries versus immediate failure.
• Fallback configuration defines alternative providers to try when retries exhaust, with ordered provider lists, per-provider retry settings, and error types that should fail immediately without attempting fallbacks.
• The Model Service provides common routing strategies as reusable patterns: cost-aware routing that tracks spending against budgets, load-based routing that distributes requests across providers, and feature-based routing that matches requests to capable providers.
• Rate limiting at the platform level protects against runaway consumption by enforcing request, token, and concurrency limits before requests reach any provider, preventing misconfigured jobs or bot attacks from generating unexpected costs that provider-side limits alone cannot catch.
• Response caching at the application level stores complete responses and returns them for identical requests at zero cost and minimal latency, effective for FAQ systems and deterministic queries with low temperature.
• Prompt caching at the provider level reduces costs when providers recognize repeated prompt content, charging discounted rates for cached tokens, particularly valuable for RAG applications with large stable document context.
• Comprehensive metrics tracking captures token usage, provider and model information, latency measurements, success and failure rates, and cache hit statistics for every request flowing through the Model Service.
• The SDK's ModelClient establishes the pattern for service clients: lazy initialization when first accessed, gRPC channel to the Gateway with routing metadata, stub from generated protobuf code, and methods that convert between Python objects and Protocol Buffer messages.
• Streaming in the SDK returns a generator that yields ChatChunk objects as they arrive from the gRPC stream, letting workflows iterate and display tokens progressively while the underlying machinery handles network details.

FAQ