2 Building with the platform: SDK and API design

The chapter explains that an AI platform only becomes useful when developers can interact with it through simple, powerful interfaces. Using a healthcare assistant as the running example, it shows that the ideal experience hides infrastructure complexity while still giving developers control over important behavior such as memory, safety, knowledge retrieval, and tool use. The SDK and APIs are presented as contracts between layers of the system: the SDK helps application developers compose AI capabilities naturally, while the external API lets other clients access those applications in a consistent way.

To make development feel effortless, the text walks through the core platform capabilities the SDK should expose: easy model access, automatic conversation memory, semantic search over organizational knowledge, configurable guardrails, external tool integration, prompt experimentation, and managed multi-step workflows. Each capability is framed around reducing friction for developers while preserving reliability, observability, and control. The workflow example shows how these services should work together so that application code reads like business logic instead of distributed systems plumbing.

The chapter then turns to deployment and communication. It argues that each workflow should run in an isolated container and be independently deployable, scalable, and versioned. External access should happen through standardized HTTP-based APIs, with synchronous, asynchronous, and streaming patterns chosen based on how long the AI task takes and how results should be delivered. Internally, the platform should use an API gateway and efficient service-to-service calls so that workflows can reach platform services without managing network details. The SDK ties all of this together by connecting to the gateway, lazily creating service clients, and capturing deployment metadata so the platform can package, route, and run workflows transparently.

Three approaches to isolating workflows: plain processes provide weak isolation with shared resources but fast startup; virtual machines offer strong isolation but require minutes to start and consume hundreds of megabytes per instance; containers provide the optimal balance with strong isolation, lightweight resource usage, and startup times measured in seconds.

Sarah’s workflow code gets packaged into a container image with all dependencies, then deployed as independently managed container replicas. The workflow service handles health monitoring, auto scaling and service registration while enforcing resource limits and security boundaries for each container instance. External clients access the workflow through the API gateway, which load balances requests across healthy container replicas.

API sprawl emerges when workflows expose inconsistent interfaces. In this example of the antipattern, 3 developer applications must each maintain custom integration code for every workflow they use, creating a tangled web of connections. Each workflow has its own authentication mechanism, response format, and conventions, forcing application developers to treat each workflow as a unique integration project rather than accessing a unified platform.

The API Gateway serves as the single-entry point for all external and internal traffic, handling authentication, rate limiting, secure connections, and health checking before routing requests to healthy containers. External clients interact with a stable API endpoint while the gateway manages the complexity of multiple container replicas, gradual version deployments, and failure detection.

The Request Flow: A user's question flows from the mobile app through the API Gateway (over HTTPS), reaches the workflow container, which then makes multiple internal gRPC calls through the gateway to reach platform services. The gateway acts as a gRPC proxy, routing internal service communication while maintaining Protocol Buffer efficiency.

The SDK provides three core capabilities: the GenAIPlatform object configures gateway connections through environment variables or explicit parameters; lazy-initialized service clients (SessionClient, ModelClient, DataClient) connect to the gateway with service-specific metadata; and the @workflow decorator captures deployment configuration that the deployment tool uses to package and register workflows with the Workflow Service. This architecture enables Sarah to write simple code that hides substantial infrastructure complexity.

When Sarah's workflow calls a platform service, the SDK lazily initializes the appropriate client on first access, establishes a gRPC channel to the gateway with service-specific metadata, constructs Protocol Buffer requests, and converts responses back to Python objects. The gateway reads the x-target-service metadata to route requests to the correct service without deserializing the binary data

• Developer experience design should prioritize immediate productivity through effortless setup, allowing developers to create working AI applications with minimal configuration while remaining extensible for advanced use cases.
• Complex multi-step workflows should feel like writing declarative business logic, with the platform handling orchestration complexity, error recovery, automatic retries, state management, and observability through the @workflow decorator pattern.
• Container-based isolation provides the right balance for workflow deployment, offering strong process boundaries, enforced resource limits, and security isolation while starting quickly and using resources efficiently compared to virtual machines or plain processes.
• One workflow equals one independently deployable service, enabling teams to deploy, scale, and version their workflows independently without coordination, even though this creates more services to manage.
• The Workflow Service acts as the operational manager for all workflow deployments, handling container lifecycle management, health monitoring, automatic scaling based on metrics, and service discovery without requiring developers to understand orchestration complexity.
• Unified workflow exposure prevents API sprawl by providing standardized domain structures where all workflows share consistent authentication, response formats, and API conventions, while developers maintain control over their specific API paths through explicit configuration.
• HTTP serves as the universal protocol for external client communication because it works everywhere—browsers, mobile apps, backend services—without requiring special libraries or setup, making it the natural choice for exposing workflows to external consumers.
• AI applications require three distinct interaction patterns over HTTP: synchronous requests for operations completing in seconds, asynchronous requests with job tracking for long-running tasks taking minutes or hours, and streaming responses using Server-Sent Events for progressive token-by-token delivery.
• The API Gateway serves as the single entry point for all external traffic, centralizing authentication, rate limiting, TLS termination, health checking, and load balancing across workflow container replicas, providing a stable interface that external clients depend on.
• Internal service communication uses gRPC instead of HTTP because it optimizes for efficiency and developer experience—making remote service calls feel like local function calls, using compact binary Protocol Buffers for 30-50% smaller payloads and 3-5x faster serialization compared to JSON.
• The API Gateway doubles as a gRPC proxy for internal communication, routing service-to-service calls while maintaining Protocol Buffer efficiency by forwarding binary data without deserialization, centralizing routing decisions while preserving gRPC's performance benefits.
• The complete request flow demonstrates architectural cohesion: external requests arrive via HTTPS at the gateway, workflows make internal gRPC calls through the gateway to platform services, and the gateway handles routing, service discovery, and connection pooling transparently.
• Lazy initialization of service clients (sessions, models, data) through Python properties ensures that unused services never get created, saving startup time and memory while making all services available through consistent platform.service_name interfaces.
• Service clients connect to the gateway using secure gRPC channels and specify their target service through x-target-service metadata headers, allowing the gateway to route requests to appropriate service containers while maintaining a single connection point.
• The @workflow decorator captures deployment configuration (name, API path, scaling parameters, response mode) as function metadata, enabling deployment tools to package workflows into containers and register them with the Workflow Service without changing function behavior.
• The SDK foundation establishes a consistent pattern that every platform service follows: connect to the gateway, specify the target service in metadata, define Protocol Buffer contracts, generate Python code, and implement methods that make gRPC calls through the gateway, creating a unified and predictable developer experience.

FAQ