Data Pipeline
Automate Reports
you own this product
This project is part of the liveProject series
Event-Driven Data Pipeline with Python and Kafka
- prerequisites
- experience with grouping records in SQL • basics of generating files
- skills learned
- end-to-end pipeline • Kafka data into the database • web scraping • creating reports
pro $24.99 per month
- access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
- choose one free eBook per month to keep
- exclusive 50% discount on all purchases
lite $19.99 per month
- access to all Manning books, including MEAPs!
team
5, 10 or 20 seats+ for your team - learn more
Look inside
As a member of the development team at Piper Data Concepts, you’ll carry out the final steps of a workflow-improvement project: enabling your client’s staff to gather workflow process information in real-time. Several prototypes have been built, and the client’s workflow is more resilient than ever. You’ll write Python code that consumes messages from Kafka and prepares them for storing in the database, create Postgres queries to access the aggregated data, and build reports in CSV files to be read by visualization tools—and ultimately, your client’s staff. When you’re done, your client’s modern system will provide a feedback loop, enable external API access to status updates, and be ready for more specialized services to be plugged in later, with no code changes.
This project is designed for learning purposes and is not a complete, production-ready application or solution.
book resources
When you start your liveProject, you get full access to the following books for 90 days.
prerequisites
This liveProject is for developers who understand the usefulness of event-driven architecture and want to learn to implement it. To begin these liveProjects you’ll need to be familiar with the following:
TOOLS- Intermediate Python
- Apache Kafka
- Faust library
- Poetry
- PostgreSQL
- Visual Studio Code
- Python library management
- Use frameworks to make project-packaging easier
- Query databases
- Create data on the fly
you will learn
In this liveProject, you’ll complete the final steps in developing an event-driven data pipeline.
- Write Python code that consumes messages from Kafka and prepares them for storing in the database
- Create Postgres queries to get the aggregates
- Build CSV reports that can be accessed by visualization tools
features
- Self-paced
- You choose the schedule and decide how much time to invest as you build your project.
- Project roadmap
- Each project is divided into several achievable steps.
- Get Help
- While within the liveProject platform, get help from other participants and our expert mentors.
- Compare with others
- For each step, compare your deliverable to the solutions by the author and other participants.
- book resources
- Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.
choose your plan
pro
monthly
annual
$24.99
$249.99
only $20.83 per month
- access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
- choose another free product every time you renew
- choose twelve free products per year
- exclusive 50% discount on all purchases
-
![]()
Automate Reports project for free
Automate Reports project for free
team
monthly
annual
$49.99
$499.99
only $41.67 per month
- five seats for your team
- access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
- choose another free product every time you renew
- choose twelve free products per year
- exclusive 50% discount on all purchases
-
![]()
Automate Reports project for free
