Advanced Stream Processing you own this product

intermediate Scala • basic Spark (DataFrame)
skills learned
perform complex arbitrary stateful computation on streams (such as handling out-of-order data and custom computation) • set up a Spark cluster • test your solution on the local cluster

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases

lite $19.99 per month

  • access to all Manning books, including MEAPs!


5, 10 or 20 seats+ for your team - learn more

Look inside

You’re the star data engineer at Free Power Corporation Limited (FPCL). The London-based power company is interested in gaining insight into its customers’ energy usage patterns, and it’s up to you to deliver a data-rich solution that satisfies the requirements of FPCL’s various teams. You’ll create a streaming Spark application to read the consumer event stream from Kafka, you’ll add information that helps the teams determine when data was generated, ingested, and processed, and you’ll write logic to reorder any late or out-of-order data. To provide vital household energy consumption statistics to the sales and electrical engineering teams, you’ll join Kafka data streams and perform complex computations on the resulting stream. To be sure your solution is ready for the teams to use, you’ll test it on the local Spark cluster. When you’ve finished, you’ll have learned advanced stream processing skills that empower you to meet the different business requirements of various enterprise departments.

This project is designed for learning purposes and is not a complete, production-ready application or solution.

project author

Gaurav Bhardwaj

Gaurav Bhardwaj has almost two decades of experience designing and developing enterprise software for large-scale data processing and machine learning. Currently working as a Big Data Architect for an IT consulting firm, he helps clients build mature data platforms (on-prem, cloud, and hybrid) and develops solutions involving large-scale data processing, data management and governance, machine learning models, and more. He has also authored official documentation for Apache HBase coprocessor.


This liveProject is for basic Scala developers and data engineers who want to learn advanced real-time data streaming skills using Apache Spark and Kafka. To begin these liveProjects you’ll need to be familiar with the following:

  • The data stream from the first liveProject in this series, Ingest Consumer Data
  • Spark
  • Command-line tools
  • Intermediate Scala knowledge
  • An IDE (such as IntelliJ, Eclipse, or VS Code)
  • Spark basics (including reading, writing, and performing simple transformations on DataFrames)


You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
book resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.

choose your plan


only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • Advanced Stream Processing project for free