Real-time Data Processing you own this product

intermediate Scala • basic Spark (DataFrame)
skills learned
connect Spark to Kafka • read the data Stream • build a streaming pipeline to perform various tasks (such as cleansing and filtering aggregation)

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases

lite $19.99 per month

  • access to all Manning books, including MEAPs!


5, 10 or 20 seats+ for your team - learn more

Look inside

As part of an endeavor to better handle surging energy prices, Free Power Corporation Limited (FPCL) has a Kafka cluster that ingests large amounts of consumer energy data. As a data engineer for FPCL, you’re already familiar with the data, so the London-based power company has tasked you with building a streaming solution that processes the data as soon as it’s available. Using Apache Spark, you’ll create an application to read the data from the Kafka streams, and you’ll save the streams to a data lake. Using a Spark API, you’ll prepare the data for analysis by performing aggregation on the fly. You’ll join the real-time stream with the static data, enriching it with customer details and enabling FPCL’s research team to gain insights about customer energy consumption patterns. When you’re done, FPCL will be better equipped to deal with rising energy costs, and you’ll have hands-on experience building a real-time data processing solution using Apache Spark and Kafka.

This project is designed for learning purposes and is not a complete, production-ready application or solution.

project author

Gaurav Bhardwaj

Gaurav Bhardwaj has almost two decades of experience designing and developing enterprise software for large-scale data processing and machine learning. Currently working as a Big Data Architect for an IT consulting firm, he helps clients build mature data platforms (on-prem, cloud, and hybrid) and develops solutions involving large-scale data processing, data management and governance, machine learning models, and more. He has also authored official documentation for Apache HBase coprocessor.


This liveProject is for basic Scala developers and data engineers who want to learn to stream and process data in real time using Apache Spark and Kafka. To begin these liveProjects you’ll need to be familiar with the following:

  • The data stream from the previous liveProject Ingest Consumer Data
  • Spark
  • Command-line tools
  • Intermediate Scala knowledge
  • An IDE (such as IntelliJ, Eclipse, or VS Code)
  • Spark basics (including reading, writing, and performing simple transformations on DataFrames)


You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
book resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.

choose your plan


only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Real-time Data Processing project for free