Managing a Distributed Electrical Grid in Real Time with Kafka

prerequisites: Advanced Java/Scala, Basics of JVM build systems, Basics of Docker, Basics of Apache Kafka.
skills learned: Fundamentals of IoT stream processing with Kafka Streams, Schema management & evolution with Apache Avro, Building distributed query engine to support real-time energy grids.

Jesse Yates

5 weeks · 6-10 hours per week · INTERMEDIATE

In this liveProject, you'll take on the role of a backend data engineer working for a rapidly growing startup. Your company installs battery packs for solar panels, as well as IoT devices monitor energy usage. These devices can help users and utilities companies better manage their energy, saving your customers money and providing clean power to the wider grid. You’ve been given a big task: build the infrastructure from the ground up that can handle the streaming events of thousands of these IoT devices. This infrastructure will be expected to scale as your company grows, and so your team has chosen to work with Apache Kafka for stream processing of all the data. Your challenges will include efficiently ingesting IoT events, managing corner cases in IoT data processing, developing fleet-wide monitoring, and providing REST services to answer questions about battery energy capacity.

This project is designed for learning purposes and is not a complete, production-ready application or solution.

book resources

When you start your liveProject, you get full access to the following books for 90 days.

project author

Jesse Yates

Jesse Yates is a Staff Engineer at Tesla and leads the development of real-time stream processing on the Data Platform Team. Prior to Tesla, Jesse created a real-time IoT data startup, helped Salesforce build the first enterprise-grade Apache HBase installation, and consulted on healthcare, defense, and cross-cloud analytic projects. He is also a committer on Apache HBase and a PMC member of Apache Phoenix, both big data storage and query projects.

Prerequisites

For developers with several years of Java or Scala experience and an understanding of the core principles of Apache Kafka. Knowledge of web-servers, databases, build systems and Docker will be helpful, but not essential.

TOOLS

Intermediate with Java/Scala
Basics of Maven, SBT, or other JVM build systems
Basics of Docker
Basics of Apache Kafka

TECHNIQUES

Basics of partitioning and hashing data
Basics of pub/sub system design

you will learn

In this liveProject, you’ll learn how to get started with stream processing and master advanced techniques around real-time data access, and streaming fault management.

Kafka Streams
Fundamentals of IoT stream processing
Advanced IoT stream processing techniques
Error handling for IoT devices
Schema management & evolution with Apache Avro
Trading off latency and throughput in Kafka
Design patterns for resilient stream processing
Evolving real-time processing for real-world use cases
Advanced partitioning schemes
Distributed, low-latency streaming queries

features

Self-paced: You choose the schedule and decide how much time to invest as you build your project.
Project roadmap: Each project is divided into several achievable steps.
Get Help: While within the liveProject platform, get help from other participants and our expert mentors.
Compare with others: For each step, compare your deliverable to the solutions by the author and other participants.
book resources: Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.

project outline

Introduction

Prerequisites Test

New module

Start Project

1. Ingesting Streaming Events and Realtime Access to Battery State

1.1 Ingesting Streaming Events and Realtime Access to Battery State

Build a Dropwizard Webserver

Avro Schema

Storing Events in a Database

Kafka Streams

Creating the Topology for the Yelling App

1.2 Submit Your Work

2. Edge Cases at Scale

2.1 Edge Cases at Scale

Canonical Data and the Schema Registry

Kafka Producers

Big Messages

2.2 Submit Your Work

3. Insight into Fleet Status

3.1 Insight into Fleet Status

Schema in Raw Topics

Streams and State

3.2 Submit Your Work

4. Helping Users Make Money

4.1 Helping Users Make Money

Joining Streams

KTables and GlobalKTables

Event Streams vs. Update Streams

Constructing the Join

Initializing the Value Transformer

The Record Stream

The Processor API

How Interactive Queries Work

4.2 Submit Your Work

5. Global Pricing and Demand/Response

5.1 Global Pricing and Demand/Response

Managing Tables and History

Building the Test

5.2 Submit Your Work

Look inside

placing your order...

Don't refresh or navigate away from the page.

Manning Early Access Program (MEAP) In MEAP, you get immediate access to a liveProject under development, so you can participate while it is created, tested, and improved. Get started today, and pick up right where you've left off whenever we update the project!

liveProject $24.99 $49.99 self-paced learning

Managing a Distributed Electrical Grid in Real Time with Kafka (liveProject) added to cart

continue shopping go to cart

continue shopping

go to cart