Three-Project Series

Become a Data Engineer with AWS you own this product

basic AWS cloud computing • advanced shell scripting • basic Python
skills learned
build ETL data pipelines using AWS Step functions • extract and process data using AWS Lambda, RDS (MySQL), AWS Glue, Amazon Athena and Redshift, AWS Kinesis • deploy pipeline resources using Infrastructure as Code (AWS CloudFormation) • build and train ML models using Amazon Personalize
Mike Shakhomirov
3 weeks · 5-7 hours per week average · BEGINNER

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases

lite $19.99 per month

  • access to all Manning books, including MEAPs!


5, 10 or 20 seats+ for your team - learn more

Step into the shoes of a data engineer working for a mobile game development studio. The company’s data architecture includes an Amazon Athena data lake and an AWS Redshift data warehouse. The board has requested data insights based on user behavior data. You’ll create data pipelines that provide improved OLAP analytics based on user engagement data, build in-app user recommendations based on purchase preferences, and implement a data-driven decision-making process.

In the first liveProject, you’ll create a batch-processing data pipeline using AWS RDS, AWS S3, and Amazon Athena to learn one of the most cost-effective data platform design patterns. Next, you’ll build a simple yet reliable data streaming pipeline that prevents resource shortages and transforms data in real-time (while it’s still relevant), ensuring more accurate data. Lastly, you’ll use Amazon Personalize to create an ML data pipeline that provides product recommendations tailored to users’ data. By the end of the series, you’ll have learned data platform design concepts, business intelligence (BI) concepts, and the extract, transform, load (ETL) process using infrastructure as code, plus you’ll have valuable firsthand experience using popular AWS data transformation and processing tools to build data pipelines.


Most of the services used in this liveProject series are available under the AWS Free Tier. However, the Free Tier doesn't cover RDS DB instances launched with Amazon Aurora, Amazon RDS for Microsoft SQL Server, or Oracle database engines. AWS RDS may incur charges if left running. Be sure to delete all associated RDS instances and backup images. Total charges should be under $2 for the series. Please check the AWS Pricing Calculator for more details and cost estimates.

These projects are designed for learning purposes and are not complete, production-ready applications or solutions.

These projects cover highly popular topics today. AWS, as a cloud platform, has a leadership position and it is very popular as an option for BI/ML/DS projects.

Ninoslav Cerkez, senior machine learning engineer, Rimac Technology

book resources

When you start each of the projects in this series, you'll get full access to the following book for 90 days.

choose your plan


only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Become a Data Engineer with AWS project for free

This course could prove beneficial for developers who are interested in branching out into the field of data engineering. Its content would provide them with a useful foundation and insight into the key concepts and techniques.

Michal Rutka, DevOps consultant, Rutka B.V

project author

Mike Shakhomirov

Mike Shakhomirov is the head of data engineering at The World's Online Festival. He has an MBA as well as a diploma in big data and social analytics from MIT, and he’s a Google Cloud Certified Professional Data Engineer. Passionate, enthusiastic, and digitally focused, he loves the challenges that the diverse gamut of digital marketing can offer. Mike is an official writer for publications including Towards Data Science and The Startup, and he’s the author of more than 50 published articles on topics such as data engineering, machine learning, and AI in digital marketing. You can find him on LinkedIn and Medium.


These liveProjects are for intermediate Python programmers who are interested in building data pipelines using AWS. To begin these liveProjects you’ll need to be familiar with the following:

  • Intermediate Python skills and knowledge
  • AWS account
  • Basic cloud computing skills
  • Basic knowledge of MySQL databases
  • Basic knowledge of serverless infrastructure
  • Build basic REST APIs
  • Deploy Lambda using the AWS CLI
  • Provision resources with Infrastructure as Code

you will learn

In this liveProject series, you’ll learn data platform design, machine learning (ML), data visualization, and business intelligence (BI) concepts, as well as how to use popular AWS data transformation and processing services to build data pipelines.

  • Shell commands and scripting to deploy your Lambda using the AWS CLI
  • Build ETL data pipelines using AWS Step functions
  • Extract and process data using AWS Lambda, RDS (MySQL), AWS Glue, Amazon Athena and Redshift, and AWS Kinesis
  • Deploy pipeline resources using Infrastructure as Code (AWS CloudFormation)
  • Visualize data and create reports and dashboards (AWS QuickSight)
  • Build and train machine learning (ML) models using Amazon Personalize


You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
book resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.