Data Pipeline

Automate Reports you own this product

This project is part of the liveProject series Event-Driven Data Pipeline with Python and Kafka
experience with grouping records in SQL • basics of generating files
skills learned
end-to-end pipeline • Kafka data into the database • web scraping • creating reports
Robert Koch and Shane Smith-Sahnow
1 week · 4-6 hours per week · BEGINNER

placing your order...

Don't refresh or navigate away from the page.
liveProject This project is part of the liveProject series Event-Driven Data Pipeline with Python and Kafka liveProjects give you the opportunity to learn new skills by completing real-world challenges in your local development environment. Solve practical problems, write working code, and analyze real data—with liveProject, you learn by doing. These self-paced projects also come with full liveBook access to select books for 90 days plus permanent access to other select Manning products. $19.99 $29.99 you save $10 (33%)
Automate Reports (liveProject) added to cart
continue shopping
adding to cart

choose your plan


only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free eBook every time you renew
  • choose twelve free eBooks per year
  • exclusive 50% discount on all purchases
  • Automate Reports eBook for free
Look inside

As a member of the development team at Piper Data Concepts, you’ll carry out the final steps of a workflow-improvement project: enabling your client’s staff to gather workflow process information in real-time. Several prototypes have been built, and the client’s workflow is more resilient than ever. You’ll write Python code that consumes messages from Kafka and prepares them for storing in the database, create Postgres queries to access the aggregated data, and build reports in CSV files to be read by visualization tools—and ultimately, your client’s staff. When you’re done, your client’s modern system will provide a feedback loop, enable external API access to status updates, and be ready for more specialized services to be plugged in later, with no code changes.

This project is designed for learning purposes and is not a complete, production-ready application or solution.

book resources

When you start your liveProject, you get full access to the following books for 90 days.

project authors

Robert Koch
Rob Koch is a Principal Data Engineer at Slalom Build, and one of the community leaders of He helps drive cloud-native architecture, blogs about migrating to the cloud and use of Lambdas, and has a passion for data- and event-driven systems. Having earned five AWS certifications (Cloud Practitioner, Big Data Specialty, DevOps Engineer Associate, SysOps Administrator Associate, and Solution Architect Associate), Robert is actively involved in the development community in Denver, often speaking at Denver Dev Day and the AWS Denver Meetup. Robert’s goal is to help the community understand the advantages of migrating to the cloud, being cloud-native, and having “serverless” applications and databases.
Shane Smith-Sahnow
Shane Smith-Sahnow is a software engineer at Netlify. He has spent time at Github and New Relic working on large scale Kafka applications, developing analytical data pipelines, and shipping large scale features.


This liveProject is for developers who understand the usefulness of event-driven architecture and want to learn to implement it. To begin these liveProjects you’ll need to be familiar with the following:

  • Intermediate Python
  • Apache Kafka
  • Faust library
  • Poetry
  • PostgreSQL
  • Visual Studio Code
  • Python library management
  • Use frameworks to make project-packaging easier
  • Query databases
  • Create data on the fly

you will learn

In this liveProject, you’ll complete the final steps in developing an event-driven data pipeline.

  • Write Python code that consumes messages from Kafka and prepares them for storing in the database
  • Create Postgres queries to get the aggregates
  • Build CSV reports that can be accessed by visualization tools


You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
book resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.