15 Building event-driven applications

Event-driven applications replace tightly coupled, request/response integrations with decoupled services that communicate by publishing and consuming events. This pattern, often powered by streaming platforms like Apache Kafka or RabbitMQ, improves responsiveness, reduces latency and batch delays, and makes systems easier to evolve. Key concepts include distinguishing event time from processing time and using time windows to bound continuous streams for accurate analytics. While streaming platforms and databases handle transport and persistence, integrating a separate stream processor can fragment the developer experience with multiple APIs, languages, and tools. The chapter positions MongoDB as a way to unify these concerns and deliver a cohesive approach to building real-time, event-driven systems.

MongoDB Atlas Stream Processing treats a continuous dataflow as an aggregation pipeline that starts with a mandatory $source and runs indefinitely, applying stateless and stateful operations to data in motion. It supports sources such as Kafka topics, MongoDB change streams (collection or database level), and even document arrays for testing, and it outputs either to streams ($emit) or to Atlas collections ($merge). Stateful processing is expressed with time windows—$tumblingWindow for non-overlapping intervals and $hoppingWindow for overlapping, sliding analysis—enabling operators like $group and $avg on bounded sets. The platform adds stream-specific stages like $validate for schema and quality checks, and adapts familiar stages such as $lookup to enrich events with Atlas data. Stream processing instances scale with workers, are managed independently of clusters, and can be created and operated via mongosh methods and the Atlas CLI, allowing developers to prototype ephemeral processors, persist named pipelines, monitor stats and samples, and integrate with existing Atlas security and operations.

For resilient pipelines, Atlas Stream Processing maintains checkpoint documents to capture progress and state so processors can resume accurately after interruptions. It also provides a dead letter queue that routes malformed, invalid, late, or erroring records to designated Atlas collections for inspection, monitoring, and optional reprocessing—often with capped collections and triggers to automate handling. Operators can define and observe DLQs, validate schemas continuously, and mix at-rest data with streams for enrichment and materialized views. Security extends Atlas’ model with a Project Stream Processing Owner role and fine-grained privilege actions (for creating, starting, stopping, listing, sampling, and getting stats), plus network controls for Atlas clusters and external sources through allowlists or VPC peering. Auditing captures authentication and management events per stream processing instance, giving teams visibility and governance across the lifecycle of their event-driven applications.

Point-to-point communication pattern, where multiple applications are interconnected through direct communication links. Each application relies on others to send or request data, creating a web of dependencies that can make the entire system fragile and resistant to change.

An event-driven system using an event streaming platform. Multiple producer applications and devices generate events, which are then sent to the event streaming platform. The platform manages these events, ensuring they are efficiently processed and delivered to the appropriate consumers. On the other side, various consumer applications subscribe to the events they are interested in, consuming the data at their own pace. This architecture allows for parallel processing and ensures that even if one part of the system fails, the rest can continue functioning smoothly, maintaining the overall system's reliability and scalability. An event streaming platform acts like the backbone that facilitates communication and event transfer between different applications and microservices. Events are stored in a message log, and message brokers, along with producer and consumer APIs, work together to ensure reliable delivery of events between independent services.

MongoDB Atlas streamlines operations with its fully managed service, allowing easy instantiation of a stream processor and database with minimal code. The platform offers a unified experience by using the same flexible data model and query API for both data in-motion and at-rest. MongoDB Atlas enhances applications by integrating event streams with database data and supports advanced stream processing, including continuous processing and validation. It also enables immediate reactions to events through serverless triggers and functions, with support for Kafka and other streaming platforms. This all-in-one approach simplifies the development and management of event-driven applications.

A stream processing instance is composed of multiple stream processors, such as "aggregation-1" and "aggregation-2," each dedicated to a specific task within the stream processing environment. These processors reside within the stream instance and are closely tied to the connection registry, which serves as a key store for managing secure connections to external resources. The bidirectional arrow between the Connection Registry and the Stream Instance signifies a dynamic interaction where the stream instance relies on the registry for credentials or configuration settings to execute processing tasks effectively.

The process flow in Atlas Stream Processing. On the left, data from Apache Kafka is ingested into Atlas, either through Kafka directly or via Atlas Change Streams, which capture real-time data changes. This data is then passed into Atlas Stream Processing, where it is continuously processed using operations such as aggregation, filtering, and routing. The diagram highlights that stateful windows are used within the processing to manage and analyze data over specific time intervals. After processing, the results are materialized into an Atlas Database for storage, or the data can be sent back to external systems like Apache Kafka for further use.

The Connect to MongoDB-in-Action-SPI interface includes steps for setting up connection security, choosing a connection method, and connecting. Under the section "Access your data through tools," you have two options. The first option is "Shell," which allows you to quickly add and update data using MongoDB's JavaScript command-line interface. The second option is "MongoDB for VS Code," which enables you to work with your data in MongoDB directly from your Visual Studio Code environment.

Add Connection interface within the Stream Processing panel in MongoDB Atlas allows you to add connections for stream processors to use for reading and writing events. You have three connection options: Kafka, Atlas Database, and Sample Stream. The form includes fields for the connection name and selecting an Atlas cluster. There's also an option to choose the execution role, such as "Read and write to any database." At the bottom, there are buttons to cancel or add the connection.

• Event-driven applications move away from the traditional model where tightly coupled services request data directly, leading to latency and complex dependencies. This traditional approach slows down the system and limits scalability.
• Traditionally, event processing is handled in two ways: embedding the logic directly into the application code, which can make the code complex and difficult to maintain, or sending events to a database for processing. The latter simplifies the code but adds latency, as events need to be stored and indexed before processing.
• Event-driven architecture decouples services by using real-time events for communication, managed by platforms like Apache Kafka. This approach reduces latency, lowers overhead, and enhances system performance.
• Today, mature event streaming platforms and databases exist for managing events, but real-time processing of high-volume streams remains challenging. The differences in languages, APIs, and tools create fragmentation, slowing down development and increasing complexity, time, and costs.
• MongoDB Atlas offers a seamless solution where events align naturally with flexible, JSON-like document structures. These events can be efficiently processed using MongoDB’s unified query API, which is central to its design.
• The Connection Registry stores configuration details for all connections between a stream processing instance and external data sources or sinks. It contains information like connection endpoints, authentication credentials, and other settings to ensure secure and reliable communication between the stream processor and external systems.
• A stream processor is a MongoDB aggregation pipeline that continuously processes streaming data from a designated source and sends the output to a specified destination.
• An Atlas stream processing instance is a designated namespace tied to a specific connection string, cloud provider, region, and optionally a security context for extra protection. Stream processors created within an instance are only available in that specific instance.
• Atlas stream processors pipelines are beginning with a $source stage that connects to a data source, initiating the flow of a continuous stream of documents. These documents must be in valid JSON or EJSON format.
• The $source aggregation pipeline stage specifies a connection in the Connection Registry to stream data from, supporting the following connection types: Apache Kafka broker, MongoDB collection change stream, MongoDB database change stream, and document array.
• The $source stage must be placed at the beginning of any pipeline where it is used, and only one $source stage is allowed per pipeline.
• The $validate stage examines streaming documents to confirm they match a defined schema, ensuring correct ranges, values, and data types are adhered to.
• The $tumblingWindow stage divides the data stream into fixed, non-overlapping windows based on a set time duration. Each document is processed only once, as each window closes before the next begins.
• The $hoppingWindow stage creates overlapping windows with a specified duration and hop interval. Documents can appear in multiple windows, providing a sliding view of the data stream.
• A document array as a data source is perfect for debugging because it lets you control static data, avoiding unpredictable live sources. This makes validating your pipeline and troubleshooting easier.
• Atlas Stream Processing uses "checkpoint documents" to track its progress in processing data. These checkpoints act like bookmarks, allowing the system to resume processing from the correct point, preventing data loss or duplication. They help maintain the state of a stream processor.
• In Stream Processing, a dead letter queue (DLQ) prevents unprocessable data from being lost by sending it to an error queue for further handling. In Atlas Stream Processing, you can configure an optional DLQ when defining a processor, with data stored in an Atlas Database Cluster and saved to a MongoDB collection.
• With Atlas Stream Processing, the existing Atlas framework for managing user permissions at the organization, project, and database levels is expanded to include roles and privileges specific to stream processing tasks. This allows for fine-grained control, and permissions can be restricted to individual stream processing instances for precise access management.

FAQ