table of content

Part 1 A database for modern web applications

1 Understanding the world of MongoDB

1.1 Examining the document-oriented data model

1.2 Scaling data horizontally

1.3 Exploring the MongoDB ecosystem

1.3.1 Learning the core MongoDB server features

1.3.2 Learning MongoDB Atlas concepts

1.4 Enhancing the TCMalloc version

1.5 Discovering MongoDB Query API

2 Getting started with Atlas and MongoDB data

2.1 Setting up your first Atlas cluster using Atlas CLI

2.1.1 Installing the Atlas CLI

2.1.2 Creating an Atlas account

2.1.3 Creating an organization

2.1.4 Creating an Atlas project

2.1.5 Creating a MongoDB Atlas cluster

2.1.6 Navigating the Atlas user interface

2.2 Loading a sample data set

2.3 Adding an IP address to the project access list

2.4 Creating a user

2.5 Establishing a connection to MongoDB through MongoDB Shell

2.6 Managing data with databases, collections, and documents

2.6.1 Working with dynamic schema

2.6.2 Working with databases

2.6.3 Working with collections

2.6.4 Working with documents

3 Communicating with MongoDB

3.1 Interacting via MongoDB Wire Protocol

3.2 Discovering mongosh

3.2.1 Connecting to MongoDB Atlas

3.2.2 Connecting to self-hosted deployments

3.2.3 Performing operations

3.2.4 Viewing mongosh logs

3.2.5 Running scripts in mongosh

3.2.6 Configuring mongosh

3.2.7 Using .mongoshrc.js

3.3 Playing with MongoDB Compass

3.4 Connecting using MongoDB drivers

3.5 Using the Node.js driver

3.6 Employing Python drivers

3.6.1 PyMongo

3.6.2 Motor

3.6.3 PyMongo vs. Motor

3.7 Integrating Ruby drivers

3.8 Learning Mongoid

4 Executing CRUD operations

4.1 Connecting to mongosh for CRUD operations

4.2 Inserting documents

4.3 Updating documents

4.3.1 Using update operators

4.3.2 Updating many documents

4.4 Updating arrays

4.4.1 Adding elements to an array

4.4.2 Removing elements from an array

4.4.3 Updating array elements

4.4.4 Updating using array filters

4.5 Replacing documents

4.6 Reading documents

4.6.1 Using logical operators

4.6.2 Using comparison operators

4.6.3 Working with projections

4.6.4 Searching for null values and absent fields

4.7 Performing regular-expression searches

4.8 Querying arrays

4.9 Querying embedded/nested documents

4.9.1 Querying on a nested field with dot notation

4.9.2 Matching an embedded/nested document

4.9.3 Querying an array of embedded documents

4.10 Sorting, skipping, and limiting

4.10.1 The sort operation

4.10.2 The skip operation

4.10.3 The limit operation

4.11 Deleting documents

4.12 Using bulkWrite()

4.13 Understanding cursors

4.13.1 Using manual iteration

4.13.2 Returning an array of all documents

4.14 Employing MongoDB Stable API

5 Designing a MongoDB schema

5.1 Organizing the MongoDB data model

5.1.1 Determining the workload of the application

5.1.2 Mapping the schema relationship

5.1.3 Applying a design pattern

5.2 Embedding vs. referencing

5.3 Understanding schema design patterns

5.3.1 Approximation pattern

5.3.2 Archive pattern

5.3.3 Attribute pattern

5.3.4 Bucket pattern

5.3.5 Computed pattern

5.3.6 Document Versioning pattern

5.3.7 Extended Reference pattern

5.3.8 Outlier pattern

5.3.9 Polymorphic pattern

5.3.10 Preallocation pattern

5.3.11 Schema Versioning pattern

5.3.12 Subset pattern

5.3.13 Tree pattern

5.4 Schema validations

5.4.1 Specifying JSON schema validation

5.4.2 Testing a schema validation rule

5.4.3 Modifying schema validator behavior

5.4.4 Bypassing schema validation

5.5 MongoDB schema antipatterns

6 Building aggregation pipelines

6.1 Understanding the aggregation framework

6.1.1 Writing an aggregation pipeline

6.1.2 Viewing the aggregation pipeline stages

6.1.3 Using $set and $unset instead of $project

6.1.4 Scenarios for $set and $unset operators

6.1.5 Scenario for the $project operator

6.1.6 Saving the results of aggregation pipelines

6.2 Joining collections

6.2.1 Creating a MongoDB view using $lookup

6.2.2 Using $lookup with $mergeobjects

6.3 Deconstructing arrays with $unwind

6.4 Working with accumulators

6.5 Using the MongoDB Atlas aggregation pipeline builder

7 Indexing for query performance

7.1 MongoDB query planner

7.1.1 Viewing query plan cache information

7.1.2 MongoDB plan cache purges

7.2 Supported index types

7.2.1 Creating single field indexes

7.2.2 Understanding compound indexes

7.2.3 Using multikey indexes

7.2.4 Using text indexes

7.2.5 Creating wildcard indexes

7.2.6 Geospatial indexes

7.2.7 Hashed indexes

7.3 Dropping indexes

7.4 MongoDB index attributes

7.4.1 Partial indexes

7.4.2 Sparse indexes

7.4.3 Time-to-live indexes

7.4.4 Hidden indexes

7.5 Understanding index builds

7.5.1 Monitoring in-progress index builds

7.5.2 Terminating in-progress index builds

7.6 Managing indexes

7.6.1 Discovering the $indexStats aggregation pipeline stage

7.6.2 Modifying indexes

7.6.3 Controlling index use with hint ()

7.6.4 Using indexes with $OR queries

7.6.5 Using indexes with the $NE, $NIN, and $NOT operators

7.6.6 Ensuring that indexes fit in RAM

7.6.7 Sorting on multiple fields

7.6.8 Introducing covered queries

7.7 When to not use an index

8 Executing multidocument ACID transactions

8.1 WiredTiger storage engine

8.1.1 Snapshots and checkpoints

8.1.2 Journaling

8.1.3 Compression

8.1.4 Memory use

8.2 Single-document transaction

8.3 Defining ACID

8.4 Multidocument transactions

8.4.1 Differentiating the Core and Callback APIs

8.4.2 Using transactions with mongosh

8.4.3 Using transactions with the Callback API

8.5 MongoDB transaction considerations

9 Using replication and sharding

9.1 Ensuring data high availability with replication

9.1.1 Distinguishing replica set members

9.1.2 Electing primary replica-set member

9.1.3 Understanding the oplog collection

9.2 Understanding change streams

9.2.1 Connections for a change stream

9.2.2 Changing streams with Node.js

9.2.3 Modifying the output of a change stream

9.3 Scaling data horizontally through sharding

9.3.1 Viewing sharded cluster architecture

9.3.2 Creating sharded clusters via Atlas CLI

9.3.3 Working with a shard key

9.3.4 Choosing a shard key

9.3.5 Using a shard-key analyzer

9.3.6 Detecting shard-data imbalance or uneven data distribution

9.3.7 Resharding a collection

9.3.8 Understanding chunk balancing

9.3.9 Administrating chunks

9.3.10 Automerging chunks

9.4 MongoDB 8.0 sharded cluster features

9.4.1 Embedding config servers in sharded clusters

9.4.2 Moving unsharded collections seamlessly between shards

9.4.3 Fragmentation

9.4.4 Faster resharding

9.4.5 Unsharding collections

9.5 Managing data consistency and availability

9.5.1 Write Concern

9.5.2 Read Concern

9.5.3 Read Preference

Part 2 MongoDB Atlas data platform

10 Delving into Database as a Service

10.1 Shared M0 and Flex clusters

10.2 Dedicated clusters

10.2.1 Atlas clusters for low-traffic applications

10.2.2 Atlas clusters for high-traffic applications

10.2.3 Autoscaling clusters and storage

10.2.4 Customizing Atlas cluster storage

10.3 Atlas Global Clusters

10.4 Going multiregion with workload isolation

10.4.1 Adding electable nodes for high availability

10.4.2 Adding read-only nodes for local reads

10.4.3 Using analytics nodes for workload isolation

10.5 Using predefined replica set tags for querying

10.5.1 Routing queries to analytics nodes

10.5.2 Isolating normal application secondary reads from analytics nodes

10.5.3 Routing local reads for geographically distributed applications

10.6 Understanding the Atlas custom Write Concerns

11 Carrying out full-text search using Atlas Search

11.1 Implementing full-text search

11.2 Understanding Apache Lucene

11.3 Getting to know Atlas Search

11.3.1 Learning Atlas Search architecture

11.3.2 Using Atlas Search Nodes

11.3.3 Atlas Search indexes

11.4 Building an Atlas Search index

11.5 Running Atlas Search queries

11.5.1 Using the $search aggregation pipeline stage

11.5.2 Executing the $searchMeta aggregation pipeline stage

11.6 Learning Atlas Search commands

11.7 Using Atlas Search Playground

12 Learning semantic techniques and Atlas Vector Search

12.1 Starting with embeddings

12.1.1 Converting text to embeddings

12.1.2 Understanding vector databases

12.2 Using embeddings with Atlas Vector Search

12.2.1 Building an Atlas Vector Search index

12.2.2 Selecting a Vector Search source

12.2.3 Defining your Vector Search index

12.2.4 Creating an Atlas Vector Search index

12.3 Running Atlas Vector Search queries

12.3.1 Querying with embeddings

12.3.2 Using prefiltering with Atlas Vector Search

12.4 Executing vector search with programming languages

12.4.1 Using vector search with JavaScript

12.4.2 Using vector search and prefiltering with Python

12.4.3 Using vector search with prefilters in Ruby

12.5 Using Atlas Triggers for automated embeddings creation

12.6 Workload isolation with vector search dedicated nodes

12.7 Improving Atlas Vector Search performance

13 Developing AI applications locally with the Atlas CLI

13.1 Introducing local Atlas clusters

13.2 Creating an Atlas cluster locally with Atlas CLI

13.2.1 Configuring Docker

13.2.2 Building your first local Atlas cluster

13.3 Managing your local Atlas cluster

13.3.1 Stopping, starting, checking, and deleting your local cluster

13.3.2 Loading a sample data set

13.4 Diving into a local Atlas cluster

13.4.1 Displaying processes

13.4.2 Executing into the container

13.5 Creating search indexes

13.5.1 Executing full-text search locally

13.5.2 Executing vector search locally

14 Building retrieval-augmented generation AI chatbots

14.1 Gaining insight into retrieval-augmented generation

14.2 Embedding LangChain in the RAG ecosystem

14.3 Introducing the MongoDB Atlas Vector Search RAG template

14.4 Getting started with AI chatbots

14.4.1 Describing LangChain capabilities

14.4.2 Starting with the LangChain CLI

14.5 Creating an AI-powered MongoDB chatbot

14.5.1 Setting up a new application

14.5.2 Inserting embeddings into MongoDB Atlas

14.5.3 Creating an Atlas Vector Search index

14.5.4 Testing a chatbot with LangServe

14.5.5 Communicating programmatically with a chatbot

15 Building event-driven applications

15.1 Understanding event-driven technology

15.2 Examining the concepts of stream processing

15.2.1 Differentiating event time and processing time

15.2.2 Using time windows

15.3 Starting with Atlas Stream Processing

15.4 Exploring Atlas Stream Processing

15.4.1 Discovering Atlas Stream Processing components

15.4.2 Understanding Atlas Stream Processing capabilities

15.5 Structuring a stream processor aggregation pipeline

15.5.1 Taking a deep dive into the $source aggregation stage

15.5.2 Using the stream processor $validate aggregation stage

15.5.3 Viewing all supported aggregation pipeline stages

15.6 Mastering Atlas Stream Processing

15.6.1 Adopting new stream processor methods

15.6.2 Using the Atlas CLI with stream processing

15.6.3 Creating your first stream processor

15.6.4 Learning the anatomy of a stream processor

15.6.5 Setting up a streams Connection Registry

15.6.6 Ensuring persistence in stream processing

15.7 Controlling the stream processing flow

15.7.1 Capturing the state

15.7.2 Using a dead-letter queue

15.8 Securing Atlas Stream Processing

15.8.1 Discovering new roles

15.8.2 Learning new privilege actions

15.8.3 Protecting network access

15.8.4 Auditing events

16 Optimizing data processing with Atlas Data Federation

16.1 Querying Amazon S3 and Azure Blob Store data via the Query API

16.2 Learning Atlas Data Federation architecture

16.3 Deploying an Atlas Federated Database instance

16.4 Limitations of Atlas Data Federation

16.5 Charges for Atlas Data Federation

17 Archiving online with Atlas Online Archive

17.1 Archiving your data

17.1.1 Seeing how Atlas archives data

17.1.2 Deleting archived documents

17.2 Initializing Online Archive

17.3 Connecting and querying Online Archive

17.4 Restoring archived data

18 Querying Atlas using SQL

18.1 Introducing the Atlas SQL interface

18.2 Connecting to the Atlas SQL interface

18.2.1 Enabling the interface

18.2.2 Accessing the interface

18.3 Querying MongoDB using SQL

18.3.1 Aggregation pipeline Atlas SQL syntax

18.3.2 Short-form Atlas SQL syntax

18.3.3 UNWIND and FLATTEN with Atlas SQL

19 Creating charts, database triggers, and functions

19.1 Visualizing data with Atlas Charts

19.1.1 Using natural language to build visualizations

19.1.2 Using billing dashboards

19.2 Atlas Application Services

19.2.1 Triggering server-side logic with Atlas Database Triggers

19.2.2 Writing Atlas Functions

Part 3 MongoDB security and operations

20 Understanding Atlas and MongoDB security features

20.1 Understanding the shared responsibility model

20.2 Managing authentication

20.2.1 Choosing an Atlas database cluster authentication method

20.2.2 Integrating with HashiCorp Vault

20.2.3 Choosing the authentication method

20.3 Handling authorization

20.3.1 Understanding the principle of least privilege

20.3.2 Differentiating Atlas user roles

20.3.3 Using MongoDB RBAC

20.4 Auditing Atlas

20.5 Encrypting data in Atlas

20.5.1 Encrypting data in transit

20.5.2 Encrypting data at rest

20.5.3 Managing encryption keys yourself

20.5.4 Encrypting during processing

20.6 Securing the network

20.6.1 Using an IP access list

20.6.2 Peering networks

20.6.3 Using private endpoints

20.7 Implementing defense in depth

21 Operational excellence with Atlas

21.1 Crafting backup strategies and practices

21.1.1 Discovering Atlas backup methods

21.1.2 Restoring an Atlas cluster

21.2 Inspecting the performance of your Atlas cluster

21.2.1 Finding slow queries

21.2.2 Improving your schema

21.2.3 Using native MongoDB diagnostic commands

21.3 Alerting and logging

21.3.1 Setting alert conditions

21.3.2 Logging in Atlas

21.4 Upgrading your Atlas cluster

Overview

16 Optimizing data processing with Atlas Data Federation

This chapter introduces Atlas Data Federation as a distributed query engine that unifies access to data across MongoDB Atlas clusters and external sources such as AWS S3, Azure Blob Storage, online archives, and HTTP/HTTPS endpoints. By defining virtual databases and collections that map to these stores, you can run complex analytical queries over both historical and live datasets using the MongoDB Query Language or SQL, all without moving or duplicating data. The approach streamlines data management, reduces ETL overhead, and supports a consistent, comprehensive view that enables faster, better-informed decisions.

The chapter details how storage configuration uses JSON mappings to bind virtual databases and collections to underlying sources, and explains the four-part architecture. Applications and services (drivers, data tools, Compass) provide user entry points; the Control Plane orchestrates requests and assembles results; an elastic Compute Plane close to data minimizes latency and movement; and the Data Plane governs secure access with role-based controls across Atlas clusters, S3, Azure Blob, and more. Beyond ad hoc queries, Federation supports copying and exporting to S3/Blob in formats like Parquet and CSV, combining data from multiple clusters and online archives, materializing aggregations across clusters and S3, and importing from S3 into Atlas.

Deployment involves creating a federated database instance in Atlas, naming it, selecting sources (entire clusters or specific collections, plus external stores), authorizing access (for example, IAM for S3), and optionally scheduling recurring exports to formats such as Parquet, CSV, BSON, or JSON. The chapter outlines limitations: supported authentication is restricted (SCRAM, X.509, OIDC, AWS IAM), there’s no monitoring for federated instances, S3 use is limited to a single AWS account, documents over 16 MB are unsupported, document order isn’t guaranteed without explicit operators, connections are capped at 60 per region with up to 30 concurrent queries, indexes cannot be created, and cursors may be terminated if they do not progress. Pricing is based on bytes processed and returned (with a 10 MB minimum per query and about $5 per TB processed), including inter-node transfers and $out/$merge writes; partitioning data and colocating Federation with S3 or Blob in the same cloud/region help manage latency and cost, and S3 can be queried only from AWS-based instances while Azure Blob requires Azure-based instances.

The Unified Query API allows to query workload from AWS S3 and Azure Blob Store.

The architecture of Atlas Data Federation.

Summary

Atlas Data Federation, a distributed query engine, enables seamless querying, transforming, and transferring of data across various MongoDB Atlas sources. It deploys federated database instances, which include virtual databases and collections mapped to your existing data stores.
Atlas Data Federation merges data from your MongoDB Atlas clusters, HTTP and HTTPS endpoints, Atlas Online Archive, and cloud storage (like AWS S3 or Azure Blob Storage) into virtual databases and collections. The data remains in its original location and format.
You can build and manage Atlas Federated Database from the Atlas UI as well as using the Atlas CLI.
Atlas Data Federation charges are incurred for data processed and returned by federated database instances. These costs are based on the volume of data handled and delivered by the system during operations.
The Atlas SQL Interface comes enabled by default upon creating a federated database instance. This allows you to start querying your data immediately without any additional configuration.

FAQ

What is MongoDB Atlas Data Federation and why would I use it?

It’s a distributed query engine in MongoDB Atlas that lets you query, transform, and analyze data across multiple sources without moving or duplicating it. You can combine MongoDB clusters, Atlas Online Archive, AWS S3, Azure Blob Storage, and even HTTP/HTTPS endpoints into a single, consistent view for analytics and decision-making.

Which data sources are supported and how do virtual databases/collections work?

Atlas Data Federation can read from Atlas clusters and online archives, AWS S3, Azure Blob Storage, and HTTP/HTTPS endpoints. You create virtual databases and collections that map to these sources via a JSON storage configuration, so you can access and query external data as if it were native.

What is the MongoDB Unified Query API and how do I query external storage?

The Unified Query API lets you use MongoDB’s Query Language (and optionally SQL via the Atlas SQL Interface) to run the same queries across Atlas clusters and external stores like S3 or Azure Blob. You can combine results from multiple sources into one result set.

How is Atlas Data Federation architected?

It consists of four components: Application & Services (tools, drivers, data science interfaces), Control Plane (orchestrates requests and aggregates results), Compute Plane (elastic compute nodes placed close to data to reduce latency and transfer), and Data Plane (where data resides, with secure, role-based access to sources like S3 and Azure Blob).

How do I deploy a Federated Database Instance?

In Atlas, go to Data Federation and click Create New Federated Database. Name the instance, add sources (Atlas clusters or external stores like S3), authorize access (for example, AWS IAM for S3), choose collections or buckets, configure when/how queries or exports run (and formats such as Parquet, CSV, BSON, JSON), review, and create. See the step-by-step guide: https://www.mongodb.com/docs/atlas/data-federation/tutorial/deploy/

What can I do beyond ad-hoc querying?

You can copy data from Atlas clusters to S3 or Azure Blob in Parquet/CSV/BSON/JSON, query multiple clusters and online archives for a consolidated view, materialize aggregation results across clusters and S3, and read/import data from S3 into an Atlas cluster.

Are there region or cloud constraints I should be aware of?

Yes. You can query AWS S3 only with federated database instances created in AWS, and Azure Blob Storage only with instances created in Azure. To avoid high data transfer costs and latency, deploy the federated instance in the same cloud provider and region as your storage.

What are the key limitations of Data Federation?

Authentication: supports SCRAM, X.509, OIDC, and AWS IAM only.
No Atlas monitoring for federated instances.
AWS S3 sources must be within a single AWS account.
Documents over 16 MB are not supported in queries.
Document order isn’t guaranteed unless you use specific ordering operators.
Up to 60 simultaneous connections per region.
Up to 30 simultaneous queries per instance.
You cannot create indexes in federated instances.

How is pricing calculated for Data Federation?

You’re billed for bytes processed and bytes returned/transferred. Processing is $5.00 per TB (rounded to the nearest MB) with a 10 MB minimum per query. Charges include data read to execute queries and data returned to clients, plus bytes transferred between query nodes and written by $out/$merge. Cloud egress/transfer rates apply (for example, AWS commonly charges $0.01/GB for in‑region returns to the client).

How can I control costs and improve efficiency?

Co-locate the federated instance with your storage in the same cloud and region, partition large files so queries read only what’s needed, and use query limits to reduce processed/returned bytes. The Compute Plane’s data-local execution also helps minimize transfer costs and latency. Note: cursors are automatically terminated if they process less than 16 MiB of results per minute.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$54.99 $27.49

you save $27.50 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more