Data Pipeline

Python and Kafka you own this product

This project is part of the liveProject series Event-Driven Data Pipeline with Python and Kafka
prerequisites
intermediate Python • beginner Docker • beginner Kafka
skills learned
Python packaging • messaging • containers
Robert Koch and Shane Smith-Sahnow
1 week · 4-6 hours per week · BEGINNER
Included with a Manning Online subscription
try now

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

Look inside

Step into the role of a developer at Piper Data Concepts (PDC), a company that specializes in helping Fortune 1000 companies improve their workflows. Your task is to review the 15-year-old workflow architecture of one of your clients, Trade Data Systems. You’ll identify issues and bottlenecks, then determine what’s needed to transform its workflow into a more modern, responsive architecture. To accomplish this, you’ll set up a development environment with Docker using Kafka, Python, and Postgres. As you go, you’ll deploy a Kafka cluster and write Python code using the Faust library to seamlessly process pre-defined business events.

This project is designed for learning purposes and is not a complete, production-ready application or solution.

book resources

When you start your liveProject, you get full access to the following books for 90 days.

project authors

Robert Koch
Rob Koch is a Principal Data Engineer at Slalom Build, and one of the community leaders of DeafintheCloud.com. He helps drive cloud-native architecture, blogs about migrating to the cloud and use of Lambdas, and has a passion for data- and event-driven systems. Having earned five AWS certifications (Cloud Practitioner, Big Data Specialty, DevOps Engineer Associate, SysOps Administrator Associate, and Solution Architect Associate), Robert is actively involved in the development community in Denver, often speaking at Denver Dev Day and the AWS Denver Meetup. Robert’s goal is to help the community understand the advantages of migrating to the cloud, being cloud-native, and having “serverless” applications and databases.
Shane Smith-Sahnow
Shane Smith-Sahnow is a software engineer at Netlify. He has spent time at Github and New Relic working on large scale Kafka applications, developing analytical data pipelines, and shipping large scale features.

prerequisites

This liveProject is for programmers interested in event-driven development who are comfortable installing server-based software. To begin these liveProjects you’ll need to be familiar with the following:

TOOLS
  • Intermediate Python
  • Apache Kafka
  • Faust library
  • Poetry
  • Git version control system
  • PostgreSQL
  • Visual Studio Code
  • Docker Desktop
TECHNIQUES
  • Python library management
  • Switching from a legacy system to a more modern
  • Reading architectural diagram of an event-driven system

you will learn

In this liveProject, you’ll identify issues and bottlenecks in a workflow and determine what’s needed to transform it to a more responsive architecture.

  • Using Python
  • Run several containers (Kafka, Zookeeper, and Postgres, using Docker Desktop)
  • Work with Postgres

features

Self-paced
You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
book resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.

choose your plan

pro

monthly
annual
$24.99
$249.99
only $20.83 per month
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Python and Kafka project for free

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Python and Kafka project for free