7 Platform Control Plane Foundations

This chapter lays the groundwork for a resilient, self-serve engineering platform by treating the platform as a product and Kubernetes as the control plane. It emphasizes automating everything behind pipelines, aiming for self-healing behavior and minimizing operational toil. Clear domain boundaries guide the implementation: account baselines, networking, customer identity, and a cloud service control plane base. Kubernetes’ declarative model and extensibility are leveraged not only to orchestrate workloads but also to enforce desired state and extend capabilities through APIs and add-ons.

The account baseline domain standardizes security, observability, and DNS across cloud accounts with recurring, idempotent enforcement and central log aggregation. It frames DNS as a product decision, establishing a platform-managed primary domain with consistent, self-serve subdomain patterns and Route53 hosted zones and delegations across accounts. Networking is handled as a dedicated transit layer that supports growth across regions and accounts, with optional CloudWAN, opinionated VPC layouts, and role-based per-cluster networks. The guidance stresses keeping platform infrastructure-as-code dedicated to the platform (not shared enterprise modules) to preserve velocity and decouple change.

Customer identity is decoupled from cloud IAM: authenticate via enterprise SSO, authorize by team membership, and propagate those team claims through an OIDC provider (e.g., GitHub teams via a SaaS IDP and device-auth flow). That identity is integrated with EKS to drive RBAC, complemented by a CLI that mints tokens and kubeconfigs for developers. The control plane base is provisioned as a vendor-managed core: EKS, an AWS-managed node group for admin services, standard add-ons (CNI, DNS, proxies, CSI drivers, pod identity), storage classes (EBS/EFS), and Karpenter for elastic, routinely refreshed compute across ARM/AMD pools. Post-apply setup (kubeconfig, secrets, namespaces, RBAC) is automated, and health is validated functionally with tests that exercise storage and dynamic node provisioning. Frequent, small releases with automated testing, routine add-on updates, safe node refreshes, and module version hygiene complete the upgrade and reliability strategy.

The importance of domain-driven design and the platform product domains. The aws-iam-profiles pipeline we created was part of the Cloud Administrative Identity product domain. We now continue to the Cloud Account Baseline domain and create the account-level baseline resources.

If Epetech were using DataDog, it is at this point that we would set up a repository and pipeline to manage the account-level integration provided by DataDog for AWS. With that integration in place, with each additional capability or feature we implement, like the networks we will provision in section 7.2, observability would be a part of the natural definition of done for the implementation.

Our engineering platform must provide a self-serve experience for each internal customer (development team) to configure their service to receive traffic based on our company’s "product" decision for how the DNS domain and subdomain names reflect our digital products.

Traffic will come into services running on the platform in a couple of different ways.

This VPC structure creates a solid foundation for most starting EKS implementations.

In an actual starting pipeline for our VPCs, we would also have the same number of VPCs as we have EKS clusters. But we typically have several more instances than we will create in our Epetech example.

For the SAAS tools that will be a part of our platform, the common enterprise SSO integration for authentication is a good starting point. Some tools, like CircleCI, can integrate directly with GitHub authentication and do not necessarily need an independent integration.

For Epetech, we would like our identity provider service to have built-in or easily configurable means of integrating with GitHub, requiring authentication through whatever means we have set up in GitHub. As a result of (1) successful authentication, we want the IDP to (2) get the list of all teams the user is a member of in our GitHub organization. Finally, we want to (3) return to the user a secure means of accessing the Platform infrastructure or custom API resources.

In this authentication and authorization flow, the IDP acts just as a secure go-between. The user must authenticate through GitHub and grant their device authorization to receive a JSON Web Token.

This configuration can be done through the Auth0 UI or programmatically. In a production environment, you should always manage configuration in code.

With vendor-managed services, we effectively decide which version of the service we want to be deployed and perhaps a handful of specific settings. Nearly all of what goes into deploying and managing the service is the cloud vendor’s responsibility.

• Establish cloud account-level security configuration early and manage within the engineering platform if the security stakeholders aren’t equipped to provide product-bound capabilities.
• Provision account-level observability dependencies early.
• A seamless and self-service experience for DNS and domain management is critical.
• Decide on a platform-managed domain naming option, and evolve from there to include custom subdomains and bring-your-own-domain capabilities.
• The left-of and right-of domain naming patterns for APIs and services are primarily a business-level product value decision.
• An API gateway may not be necessary unless supporting third-party developers; focus on zero-trust network patterns and internal API management.
• Set up release pipelines for DNS configurations and account-level resources to ensure consistent deployment across environments.
• Design a cloud-vendor-managed transit network that makes adding networks a low-complexity task.
• Zero-trust networking done right can simplify the execution of business decisions to make internal resources available to customers or third-party partners.
• Implement a role-based network structure where each Kubernetes cluster has a dedicated VPC, named according to the cluster for easy future scalability.
• Provision VPCs and subnets in specific regions with designated IP spaces to support different roles, such as nodes, databases, and ingress.
• Make platform customer identity its capability within the engineering platform architecture as a key means of providing flexibility in creating user experiences and supporting evolutionary architecture - this is one of those decisions you will wish you got right at the start.
• Use a SaaS Identity Provider like Auth0 to provide a standards-based security protocol and act as the provider between an authoritative source of authentication and the source of authorization claims.
• The OAuth2 OIDC device-auth-flow is an adequate standard for users of the platform to generate short-lived credentials for accessing platform infrastructure and custom services from their laptops.
• The primary permission boundary (user claim) should be team membership. This maps well to domain-bound team topologies, and when assumed to be the central goal in all the RBAC capabilities, is more likely to result in the most effective implementation the first time.
• Create a dedicated pipeline for orchestrating the cloud provider-managed aspects of the kubernetes control plane.
• Technologies like Karpenter provide more efficient means of maintaining short-lived nodes and node pools comprised of an efficient mix of node sizes and attributes.
• Cloud-provided storage classes provide a vendor-managed solution for many everyday attached storage needs.
• Integrate kubernetes directly with your identity provider solution to provide users a direct means of interacting with the kubernetes API.
• Include automated collection of kubernetes configuration details in the control plane base pipeline.
• Integration testing of the EKS pipeline includes deploying test applications that utilize the features in a customer-like manner to confirm the actual implementation health.
• Arm nodes on most cloud providers offer a more performant and cost-effective option.
• A platform CLI provides an effective touchpoint for users to interact with platform APIs. Whether creating CLI or UI touchpoints, the service interface (API) always comes first.

FAQ