Why Run MongoDB on Everpure FlashArray?
Page content
I’ve spent a lot of years around SQL Server on shared storage, and lately I’ve been spending that time with MongoDB. The database engine couldn’t be more different, but the storage conversation sounds very familiar. How do I protect it? How do I grow it? How do I get another copy of it somewhere else, and how long is all of that going to take?
This is the first post in a series where I run MongoDB replica sets on Everpure FlashArray in my lab, with the commands and the real output in every post. In this post, I’m going to show you why you’d run MongoDB this way at all: what MongoDB asks of storage, where disks inside every node start to hurt, what changes when the volumes come from a FlashArray, and a map of the rest of the series. There’s no lab output in this one, just the reasoning. Let’s go.
What MongoDB Asks of Storage
A MongoDB deployment is built from mongod processes organized into replica sets. Each data-bearing member keeps its own full copy of its replica set’s data. A sharded cluster splits the data across several replica sets, called shards, with mongos routers and config servers coordinating access.
From a storage point of view, each data-bearing member keeps its own copy of the data, the indexes, the journal, and the oplog. WiredTiger uses checkpoints and a write-ahead journal for durability and crash recovery. The oplog records every change so the other members can replicate it, and in a replica set it has a second job: after a crash, mongod replays it to bring your collections forward from the last stable checkpoint.
A three-member replica set means three copies of the same data. Shard it, and every shard is a replica set with its own copies. The storage you need depends on how the data is distributed and replicated, not just the logical size of the database. And where those copies live decides which failures the cluster can ride out.
The Problem with Disks in Every Node
The default way to run self-managed MongoDB is local disks in each server. It works, and it’s simple on day one. Over time, a few things start to hurt:
- Stranded capacity: You buy disk node by node. Some nodes fill up while others sit half empty, and you can’t lend free space from one server to another.
- Compute and storage move together: If a server dies, its data is stuck inside it until you repair or rebuild that server. If you need more disk, you often end up buying more server to get it.
- Backup is its own project:
mongodumpis a logical backup. It reads every document and writes it out somewhere else, and a restore has to insert every document again and rebuild every index. It’s a great tool, but both directions take longer the more data you have. - Rebuilding a member is slow: A brand new member, or one that’s fallen too far behind, does an initial sync. It copies every document over the network and rebuilds every index, so the time grows with your data, and the member it syncs from carries that read load the whole time.
- Local storage is only as reliable as the server it’s in: A failed drive, RAID controller, a firmware update that needs a reboot, or a motherboard that dies all take the member down with it. RAID inside the server helps with a single drive, but the rebuild runs on the same box that’s serving your database, and everything else in that server is still a single point of failure.
That last one is worth sitting with. A replica set is built to survive losing a member, and it does. But with local disks, every hardware problem on a node becomes a lost member. While that node is out, you’re running on one less copy of your data, and getting it back usually means a rebuild and an initial sync. The replica set’s redundancy gets spent on disk failures, when it’s really there for losing a whole node or a whole site.
None of this is a flaw in MongoDB. It’s what happens when the storage lives inside the node.
What Changes on Everpure
MongoDB itself doesn’t change. The mongod processes, the replica sets, and the oplog behave exactly the same. What changes is that each member’s volumes come from a shared FlashArray instead of a disk inside the node, and that opens up a handful of things. Most of them are what I’m going to show you in this series:
- Snapshots that don’t freeze writes: A protection group snapshot is atomic on the array. Put a member’s data, journal, and oplog volumes in one protection group, and its snapshot is crash-consistent without pausing writes. WiredTiger is built to recover from a crash, and that’s exactly what a restored snapshot looks like to it. Put all three members in one group on one array, and one call captures the whole replica set. In the single-array post, the replica set kept doing about 2,700 operations a second right through the snapshot.
- Restore by overwriting a volume: Restoring a volume from a snapshot is a metadata operation on the array, not a copy through your host, so the array’s part takes the same time whether the database is 10 GB or 10 TB. What’s left is
mongodshutting down, recovering, and electing a primary, and that’s most of your wait. Every restore in this series is built on this. - Replacing the initial sync with a snapshot: Instead of copying every document over the network, you replicate a snapshot to the member’s array, overwrite its volumes, and let it catch up from the oplog. After the first transfer, the array sends only the blocks that changed. In the initial sync post, I restore a member from its own snapshot, show you what happens when it’s outside the oplog window, and rebuild every secondary from one snapshot of the primary, taken under load with no lock.
- Growing as a fleet: With Everpure Fusion, you connect to one array and manage the whole fleet from there. Adding a member becomes a placement question, not a storage project. In the Fusion post, I describe a member as a workload preset, let Fusion pick the array, and seed the new member on a fourth array from a replicated snapshot.
- A copy off the FlashArrays: FlashBlade gives you somewhere else to keep a logical backup, on a system that isn’t holding the live data. In the FlashBlade post, I protect a
mongodumpwith a file system snapshot and restore straight from it. - Hardware failures stay below the database: Everpure’s measured availability across its install base is 99.9999%, including maintenance and component failures. For MongoDB, that changes what a hardware failure means. A failed drive or controller in the array is something the array handles while it keeps serving I/O, so the member on top of it never goes down. The replica set’s redundancy goes back to its real job: surviving the loss of a server or a site, not a disk.
- Array upgrades don’t move your data: Controllers and flash media get replaced in place, so you don’t plan a database migration around an array upgrade.
Number 1 has a boundary worth knowing up front. The crash-consistent snapshot is per replica set member: a mongod’s data, journal, and oplog volumes have to be in the same protection group, so they’re captured at the same instant. Get that right, and over the course of this series you’ll learn exactly when you need fsyncLock and when you don’t. I think you’ll be surprised how rarely you need it. Restoring a member, rebuilding one from a snapshot of the primary while it’s under load, even putting a whole replica set back from one snapshot: none of those need a lock. The fsyncLock post covers the one replica set case that does, and my post on app-consistent MongoDB snapshots across multiple FlashArrays covers sharded clusters.
What’s in This Series
This series covers MongoDB replica sets, not sharded clusters. Sharding brings its own rules for backups, so it gets its own series next. Starting with replica sets keeps the moving parts visible.
I’m building this in a small lab with seven MongoDB nodes, four FlashArrays, and a FlashBlade, all in one Fusion fleet. The code is Python, short and linear, so you can read it top to bottom, and every post has the code and the real output.
- Snapshot recovery on a single FlashArray: A three-member replica set on one array, with each member’s data, journal, oplog, and log on separate volumes. I take a protection group snapshot under load, drop a database, and restore every member by overwriting its volumes.
- Replacing the initial sync with snapshots: Restoring one member from its own local snapshot inside the oplog window, why that means taking snapshots on a schedule, and what happens outside the window. Then rebuilding a member, and every secondary, from one replicated snapshot of the primary, with no lock.
- When you need fsyncLock: A few seconds of
fsyncLockon the primary, one protection group snapshot per array, and a restore of every member from its own array’s local snapshot, back to the exact count I recorded under the lock. I also show you what goes wrong without it. - Expanding with Fusion: Adding a fourth member on a fourth array with a workload preset and a placement recommendation, then seeding it from a replicated snapshot of an existing member, with no initial sync.
- FlashBlade alongside the cluster: A
mongodumpfrom a secondary onto a FlashBlade file system, protected by a file system snapshot, and a restore straight from that snapshot into a scratch database. - A tag-based snapshot catalog: Keeping the catalog of snapshot sets in the snapshot tags themselves, so a restore of every member across the fleet needs one value and checks its own result.
Terms in This Series
This series sits between two teams, so here are the terms each side uses, in plain words. If you run the storage, start with the MongoDB list. If you run MongoDB, start with the storage list.
MongoDB Terms, for Storage Admins
- Replica set, primary, secondary: A group of
mongodprocesses that each hold a full copy of the same data. One member, the primary, takes the writes. The secondaries copy every change from it. - Oplog: The ordered log of every change to the data, stored in the
localdatabase. Every member keeps its own: the primary writes it, and the secondaries copy its entries and apply them to stay current. It has a configured size, so the oldest entries roll off as new ones arrive. - Oplog window: How far back the oplog reaches, measured in time. A member that comes back while its last operation is still in another member’s oplog catches up by replaying what it missed. Once that operation has rolled off every member it could sync from, it can’t.
- Initial sync: How MongoDB builds a member from nothing. It copies every document over the network and rebuilds every index, so the time it takes grows with the size of the data.
- opTime: A position in the oplog. Two members at the same opTime have applied exactly the same changes.
- dbHash: A hash of a database’s contents, computed on one member. If two members report the same
dbHash, they hold the same data. - Journal: WiredTiger’s write-ahead log. In a replica set it covers the
localdatabase, the oplog included. After a crash, or a restore from a snapshot, WiredTiger opens the last stable checkpoint and replays the journal, and thenmongodreplays the oplog to bring your collections forward. - fsyncLock: A command that flushes pending writes to disk and then blocks new writes on that
mongoduntilfsyncUnlock. Reads keep working. Lock the primary, and you’ve stopped every write to the replica set. - Election and priority: When there’s no primary, the members vote one in. A majority of the voting members has to agree. Once a member with a higher priority is caught up, it takes over as primary.
Storage Terms, for MongoDB DBAs
- Volume: A block device on a FlashArray, presented to a host. In this series each
mongodhas separate volumes for its data, journal, oplog, and log. - Protection group: A set of volumes that get snapshotted together. A protection group snapshot is atomic: every volume in the group is captured at the same instant, like pulling the power on all of them at once.
- Snapshot and overwrite: A snapshot is a point-in-time copy of a volume. Overwriting a volume from a snapshot puts that copy back in place. The device, its serial, and the host connection stay the same, and on the array it’s a metadata operation, not a data copy.
- Replication connection and target: Two arrays can be connected for replication. A protection group with another array as its target sends its snapshots there, and after the first one, only what changed since the last replicated snapshot.
- Everpure Fusion fleet, gateway, and
context_names: Arrays joined into a fleet can be managed through one of them, the gateway. You log in once, and each API call names the array it’s for withcontext_names. In my lab, snapshot tags were the one exception, and the catalog post shows how I handled it. - Preset, workload, and placement: A Fusion preset is a template for a set of volumes. A workload is what you deploy from it, and a placement recommendation is Fusion telling you which array it fits on best.
- Volume serial and WWN: Every FlashArray volume has a serial, and the host sees it inside the device’s WWN, after the
624a9370prefix. That’s how the code in this series matches a Linux device to the volume behind it. - Crash-consistent and app-consistent: A crash-consistent copy looks like the moment the power went out, which MongoDB is built to recover from. An app-consistent copy is taken while the application has paused its writes, so every member is at one known point.
Wrapping Up
MongoDB needs copies of your data in the right places. Shared storage lets you choose where those copies live, and it gives you a fast way to make, move, and restore them. The bottom line is that the array’s part of a snapshot restore doesn’t grow with your data, and an initial sync or a logical restore does. The rest of this series is me doing these jobs on real gear, and showing you where it gets complicated.
Two things to keep in your head before the lab posts. First, the no-lock snapshot works per member: a mongod’s data, journal, and oplog go in one protection group. Second, when every member restores from its own array’s local snapshot and those are separate snapshots on different arrays, you need a few seconds of fsyncLock on the primary, and a sharded cluster needs MongoDB’s full procedure. MongoDB’s documentation on replication and backups with file system snapshots is good background for the whole series. Next up is the single-array post, where I snapshot a replica set under load and bring it back after a dropped database. Follow along, try it in your own lab, and let me know how it works in your environment.
MongoDB on Everpure Series
- Why Run MongoDB on Everpure FlashArray? (This Post)
- Snapshot Recovery of a MongoDB Replica Set on a Single FlashArray
- Replacing the MongoDB Replica Set Initial Sync with FlashArray Snapshots
- When You Need fsyncLock: Restoring a MongoDB Replica Set from Local Snapshots on Every FlashArray
- Expanding a MongoDB Replica Set with Everpure Fusion
- Where FlashBlade Fits When MongoDB Lives on FlashArray
- A Tag-Based Snapshot Catalog for a MongoDB Replica Set Across FlashArrays with Everpure Fusion
Related: App-Consistent MongoDB Snapshots Across Multiple Pure Storage FlashArrays, my earlier post on snapshotting a sharded MongoDB cluster across two arrays.