As part of an endeavor to better handle surging energy prices, Free Power Corporation Limited (FPCL) has a Kafka cluster that ingests large amounts of consumer energy data. As a data engineer for FPCL, you’re already familiar with the data, so the London-based power company has tasked you with building a streaming solution that processes the data as soon as it’s available. Using Apache Spark, you’ll create an application to read the data from the Kafka streams, and you’ll save the streams to a data lake. Using a Spark API, you’ll prepare the data for analysis by performing aggregation on the fly. You’ll join the real-time stream with the static data, enriching it with customer details and enabling FPCL’s research team to gain insights about customer energy consumption patterns. When you’re done, FPCL will be better equipped to deal with rising energy costs, and you’ll have hands-on experience building a real-time data processing solution using Apache Spark and Kafka.