Three-Project Series

Become a Data Engineer with AWS you own this product

prerequisites
basic AWS cloud computing • advanced shell scripting • basic Python
skills learned
build ETL data pipelines using AWS Step functions • extract and process data using AWS Lambda, RDS (MySQL), AWS Glue, Amazon Athena and Redshift, AWS Kinesis • deploy pipeline resources using Infrastructure as Code (AWS CloudFormation) • build and train ML models using Amazon Personalize
Mike Shakhomirov
3 weeks · 5-7 hours per week average · BEGINNER

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • share your subscription with another person
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Step into the shoes of a data engineer working for a mobile game development studio. The company’s data architecture includes an Amazon Athena data lake and an AWS Redshift data warehouse. The board has requested data insights based on user behavior data. You’ll create data pipelines that provide improved OLAP analytics based on user engagement data, build in-app user recommendations based on purchase preferences, and implement a data-driven decision-making process.

In the first liveProject, you’ll create a batch-processing data pipeline using AWS RDS, AWS S3, and Amazon Athena to learn one of the most cost-effective data platform design patterns. Next, you’ll build a simple yet reliable data streaming pipeline that prevents resource shortages and transforms data in real-time (while it’s still relevant), ensuring more accurate data. Lastly, you’ll use Amazon Personalize to create an ML data pipeline that provides product recommendations tailored to users’ data. By the end of the series, you’ll have learned data platform design concepts, business intelligence (BI) concepts, and the extract, transform, load (ETL) process using infrastructure as code, plus you’ll have valuable firsthand experience using popular AWS data transformation and processing tools to build data pipelines.

Pricing

Most of the services used in this liveProject series are available under the AWS Free Tier. However, the Free Tier doesn't cover RDS DB instances launched with Amazon Aurora, Amazon RDS for Microsoft SQL Server, or Oracle database engines. AWS RDS may incur charges if left running. Be sure to delete all associated RDS instances and backup images. Total charges should be under $2 for the series. Please check the AWS Pricing Calculator for more details and cost estimates.

These projects are designed for learning purposes and are not complete, production-ready applications or solutions.

These projects cover highly popular topics today. AWS, as a cloud platform, has a leadership position and it is very popular as an option for BI/ML/DS projects.

Ninoslav Cerkez, senior machine learning engineer, Rimac Technology

here's what's included

Project 1 Data Pipeline with Amazon Athena

Congratulations! You’ve just been hired as a data engineer for a mobile game development studio. The company’s modern data platform architecture includes an Amazon Athena data lake and an AWS Redshift data warehouse solution. Your task is to enable batch processing of revenue transaction data by creating an end-to-end data pipeline, connecting various data sources—including user engagement events, stage controls, and public chat messaging—to the lake house solution. Using AWS CloudFormation, you’ll provision the resources required for the data pipeline. You’ll connect a MySQL data source to the AWS S3 Data Lake and transform data in the data lake using Amazon Athena. You’ll wrap up the project by creating a dynamic analytics dashboard with AWS QuickSight. When you’re done, you’ll have built a batch-processing data pipeline, start to finish, using Amazon Athena.

Project 2 Data Streaming in AWS

As a data engineer for a mobile game development studio, your task is to create a data streaming pipeline that collects and processes large streams of data records in real-time for lightning-fast analytics. Your company’s modern data platform architecture includes an Amazon Athena data lake and an AWS Redshift data warehouse solution. To store files, you’ll create an AWS S3 bucket, and you’ll create an AWS Kinesis delivery stream by using the boto3 library to connect to AWS Kinesis endpoints and send event data to the service. You’ll provision AWS Redshift resources and connect them to your AWS Kinesis Data Stream to analyze user behavior data to understand the user's journey inside the app. When you’re done, you’ll have a simple yet reliable data streaming pipeline that prevents resource shortages and transforms data in real-time—while it’s still relevant—ensuring more accurate data.

Project 3 ML Pipeline with Amazon Personalize

Help your company’s messenger application provide better product recommendations for its customers. As a data engineer at the company, your task is to create a machine learning (ML) pipeline using the Amazon Personalize service. You’ll use CloudFormation templates to create a repository for the required AWS infrastructure resources, and AWS Glue to transform the raw user engagement data. Using Amazon Personalize, you’ll import a dataset and create and train the Amazon Personalize ML model for your users’ recommendations. To complete the project, you’ll create a workflow to train your Amazon Personalize recommendation solution using AWS Step Functions and user engagement events. When you’re done, you’ll have designed an ML pipeline using the Amazon Personalize API that provides product recommendations that suit your users best.

book resources

When you start each of the projects in this series, you'll get full access to the following book for 90 days.

choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Become a Data Engineer with AWS project for free

This course could prove beneficial for developers who are interested in branching out into the field of data engineering. Its content would provide them with a useful foundation and insight into the key concepts and techniques.

Michal Rutka, DevOps consultant, Rutka B.V

project author

Mike Shakhomirov

Mike Shakhomirov is the head of data engineering at The World's Online Festival. He has an MBA as well as a diploma in big data and social analytics from MIT, and he’s a Google Cloud Certified Professional Data Engineer. Passionate, enthusiastic, and digitally focused, he loves the challenges that the diverse gamut of digital marketing can offer. Mike is an official writer for publications including Towards Data Science and The Startup, and he’s the author of more than 50 published articles on topics such as data engineering, machine learning, and AI in digital marketing. You can find him on LinkedIn and Medium.

Prerequisites

These liveProjects are for intermediate Python programmers who are interested in building data pipelines using AWS. To begin these liveProjects you’ll need to be familiar with the following:

TOOLS
  • Intermediate Python skills and knowledge
  • AWS account
  • Basic cloud computing skills
  • Basic knowledge of MySQL databases
  • Basic knowledge of serverless infrastructure
TECHNIQUES
  • Build basic REST APIs
  • Deploy Lambda using the AWS CLI
  • Provision resources with Infrastructure as Code

you will learn

In this liveProject series, you’ll learn data platform design, machine learning (ML), data visualization, and business intelligence (BI) concepts, as well as how to use popular AWS data transformation and processing services to build data pipelines.

  • Shell commands and scripting to deploy your Lambda using the AWS CLI
  • Build ETL data pipelines using AWS Step functions
  • Extract and process data using AWS Lambda, RDS (MySQL), AWS Glue, Amazon Athena and Redshift, and AWS Kinesis
  • Deploy pipeline resources using Infrastructure as Code (AWS CloudFormation)
  • Visualize data and create reports and dashboards (AWS QuickSight)
  • Build and train machine learning (ML) models using Amazon Personalize

features

Self-paced
You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
book resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.