6 Building a Software-Defined Engineering Platform

This chapter turns prior principles into practice by guiding the first steps of building a software-defined engineering platform for the fictional company Epetech. It frames the problem—developer friction from silos, ticket-driven operations, and rising incidents—and sets a product mindset goal: a self-serve platform that lets teams design, build, release, and operate with speed, quality, resilience, and compliance. The authors advocate starting small in a greenfield approach, leaning on high‑quality SaaS tools to solve the bootstrap paradox (version control, secrets, state, and pipelines) while selecting tools with clear criteria: strong APIs, alignment with identity and access models, pipeline-optimized CLIs, service accounts/OIDC support, a good UI, and a single source of truth for sensitive configuration.

The chapter defines how platform work flows through pipelines at three scopes: account-level (once per cloud account), control-plane-level (per Kubernetes cluster instance), and namespace-level (application paths to production). It introduces a Preview cluster to safely test platform changes against real workloads without blocking teams. A set of core engineering practices anchor all infrastructure pipelines: test-driven development, trunk-based workflows, a fixed path to production, keeping code DRY across environments, minimal CI-stored secrets (pull from a central secrets manager at runtime), and fully declarative IaC (Terraform). Quality is enforced with static analysis and commit hooks, and pipelines follow a “dumb pipelines, smart code” model that emphasizes reusable, versioned libraries (e.g., orbs/actions) with injection points and overrides to balance standardization with team autonomy. Fast, ephemeral runners—hosted or private—are encouraged to cut wait time, support debugging, and persist artifacts where needed.

Finally, the chapter operationalizes cloud administrative identity: a clear account/project structure (a top-level “state” domain, platform sandbox, nonproduction, and production), least-privilege IAM roles matched to pipelines, and service accounts whose credentials are centrally managed. It walks through bootstrapping the first IAM roles and service accounts, wiring pipelines to assume roles, and storing initial credentials in a secrets vault so automation can take over. Robust operations are built in with a dual-key rotation pattern to avoid race conditions, scheduled integration tests to detect configuration drift, semantic-tagged release workflows with approvals, and optional, actionable notifications and automated release notes. The outcome is a sustainable, evolvable platform foundation that accelerates delivery, improves reliability, and reduces organizational friction.

We want to create an internal product that provides a genuinely self-service experience where developers can imagine, design, build, release, and operate their applications with agility, high engineering quality, greater operational resiliency, and confidence in meeting compliance requirements, yet without all the engineering friction they usually experience. In other words, we are creating an Engineering Platform.

The engineering platform product domains also show the basic dependencies and almost exactly the order in which we'll set up the initial product infrastructure pipelines. This diagram shows the pipelines we will build in each domain to create our Epetech starting engineering platform.

Resources we will need to begin building our engineering platform product foundation.

Regardless of which tool we are examining, there are general criteria we should carefully consider in determining whether or not the tool will integrate effectively and help create the experience we want for our platform.

We are using the same tools we will integrate with the platform for our customers’ use. To create the truly self-service experience we want, beyond the general criteria, we need the following from our secrets management tool.

Our pipeline tool will determine the quality of the continuous integration and delivery experience and needs specific additional attributes beyond the general selection criteria.

We will be using AWS in our Epetech example; some things we manage will be Account-level configurations. If we were using Google Cloud, these would be project-level configurations. Each cloud vendor will have some analogous structure.

We want instances of our platform that we can use to develop and test new features or functionality before developers have access. We also want separate instances of our platform to provide developers with their pre-production environments.

For users of the platform, an environment in the path-to-production is a kubernetes namespace. There will be one or more namespaces in a pre-production cluster, or eventually multiple clusters, where development teams test their application before deploying to a production cluster. Something you may not have seen before is the Preview environment in the similarly named Preview cluster.

Well-architected Infrastructure-as-code pipelines are no different from well-architected regular software pipelines. This is why we prefer the term 'software-defined infrastructure' to differentiate between simply using an IaC framework like Terraform and the software engineering practices that are a part of sustainable, high-quality software developer lifecycle management of an engineering platform product.

Write the tests. Run the tests. The tests are failing at this point. Then, start writing and applying the infrastructure code until the tests pass. Finally, refactor the code for dryness, readability, performance, etc.

So far, in this section, we’ve talked about seven essential infrastructure pipeline practices, discussed how to choose an IaC framework, and done a deeper dive into test-driven development of infrastructure code and performing static code analysis. Now, let’s discuss reusable pipeline code.

We intentionally want to constrain the complexity of our pipelines to these fundamental structures.

In CircleCI, you list all the jobs you want the pipeline to run beneath the workflows key. You can refer to a job that has been defined right in the same config.yml file, or you can refer to a job defined in an Orb that will be pulled from the orb registry at pipeline runtime.

Specific tools or actions are the most basic type of shared pipeline code. But don't stop there.

An Engineering Platform exists within your organization’s cloud structure, like any software product. This diagram shows a typical, well-formed AWS account structure with common OUs and clear product domain boundaries.

The first pipeline we create will manage the roles that our service accounts will assume to enable a pipeline to make the desired changes. Roles will be the same across all the engineering platform product accounts. In our example, we will also manage the IAM User service accounts. These exist only in a single AWS account.

Let’s walk through the logic of the first stage in our pipeline. We want to take these steps whenever a change is pushed to this repository.

Our service account has two valid credentials at any given time. Only the latest credential is maintained in the secrets store, and the prior keys remain valid until the next rotation to prevent workloads and orchestration from failing due to the race condition of the key being changed while a job is already in process.

In the four-account strategy described at the start of this chapter, our service accounts are deployed to the STATE account only, and it is also in this account where we perform the initial configuration test of the Roles.

• Start with a simple system that evolves into a complex system.
• Traditional IT silos and DevOps models have limitations in achieving platform goals.
• Advocate for a unified team approach in a greenfield setting to manage domain boundaries effectively.
• Effective product ownership and platform engineering practices are essential for long-term success.
• Tools must align with effective platform engineering practices and customer needs.
• High-quality, secure SAAS tools are preferred for long-term efficiency.
• Tools should be easy to integrate, support pipeline use, and provide a single source of truth for secure values.
• Encourage developer ownership of pipeline code to minimize manual intervention.
• Optimize pipeline performance and provide secure debugging access for developers.
• Define platform engineering practices for well-architected infrastructure pipelines.
• Focus on continuous deployment automation, secrets management, and avoiding duplicated code.
• Adopt declarative Infrastructure-as-Code (IaC) frameworks like Terraform for cloud-native architectures.
• Ensure pipelines are versioned, shared, and optimized for maintainability and scalability.
• Infrastructure is highly declarative, with actions happening inside vendor APIs.
• Integration tests for infrastructure code are valuable, even though they do not directly write the action code.
• Integration tests are crucial for confirming reusable modules and providing the intended resources.
• Tests can demonstrate story acceptance criteria, like naming conventions or specific configurations.
• Recurring tests help detect configuration drift caused by manual changes outside the pipeline.
• Writing tests first clarifies desired outcomes and reduces the risk of flawed test logic.
• Use tools like AWSpec and InSpec for infrastructure configuration testing.
• Static code analysis is vital for maintaining IaC code quality, focusing on syntax, style, and security.
• Tools like Tflint and Trivy validate best practices and identify security issues in Terraform code.
• Shared pipeline code libraries (e.g., CircleCI Orbs) should allow flexibility for developers.
• Pipelines should focus on triggers and event order, with specifics handled by shared code.
• Injecting custom steps, overriding parameters, and allowing tool version overrides improve pipeline flexibility.
• Managing shared pipeline resources effectively reduces long-term challenges in large organizations.
• Users should be encouraged to submit pull requests, but can also clone and modify Orbs as needed.
• Building and maintaining reusable pipeline code requires applying software development principles.
• Private executors (runners) handle pipelines in internal, private networks, ensuring security and efficiency.
• Manage cloud organization structure, identities, and permissions for service accounts in foundational infrastructure automation.
• A well-organized cloud structure should align with business and technology needs.
• A top-level domain account is needed for product-wide configuration and governance policies.
• Platform engineering teams need dedicated test environments, often called Sandbox accounts, separate from customer-facing environments.
• Separate non-production and production environments are crucial for governance and stability.
• Service accounts and roles in AWS or similar providers require specific permissions defined by Roles, which are assumed by pipeline service accounts.
• Infrastructure pipelines need dedicated roles for managing specific resources, even if only a few exist.
• Initial service account setup involves bootstrapping with minimum configuration before pipelines can manage resources autonomously.
• Use Terraform with best practices like version pinning, role path organization, and consistent tagging.
• Not every developer wants or needs to learn Terraform or understand cloud account structures. Tailor platform abstractions to match team expertise and reduce cognitive overload.
• Just because your organization is structured a certain way doesn’t mean that structure should be visible (or enforced) across environments or to every developer. Keep internal complexity hidden where possible.

FAQ