Where FlashBlade Fits When MongoDB Lives on FlashArray
Page content
Every copy of MongoDB data I’ve made so far in this series lives on a FlashArray. The snapshots sit on the same array as the volumes they protect, and even the replicated ones land on another FlashArray as the same kind of block copy. That’s exactly what you want for fast recovery. It’s also the thing that nags at me. If every copy you have lives on the same kind of system as the live data, a bad day for that system is a bad day for your backups too. So I wanted a copy somewhere else, in a format that doesn’t care what storage produced it.
That’s where FlashBlade comes in. Before anyone asks, no, I’m not putting MongoDB’s data files on it. The dbPath belongs on block storage, and in this series that’s FlashArray. FlashBlade’s job is the copies around the cluster, and the first one I reach for is a logical backup. In this post, I’m going to show you how I run mongodump from a secondary onto an NFS file system on FlashBlade and take a file system snapshot of the dump. Then I delete the dump on purpose and restore straight out of the snapshot. The terms list in the intro covers the replica set and snapshot basics, and I’ll explain NFS export policies and the mongodump options as they come up. Let’s go.
The Problem We’re Solving
The protection group snapshots in the earlier posts are physical copies. They’re fast, and they bring back a whole member or a whole replica set straight from the array. But there are three things they don’t do well:
- They share a home with the data: A snapshot lives on the array that holds the volumes. Replication moves a copy to another array, but it’s still a block copy on a FlashArray.
- They’re all or nothing: Overwriting a volume rewinds everything on it. If someone needs one database back, or wants to look at yesterday’s data next to today’s, a volume overwrite is the wrong tool.
- They’re tied to the storage: A volume snapshot is only useful to something that can mount that volume. A dump is a directory of BSON files that any
mongorestorecan read.
A logical backup fixes all three, and it has its own cost. Restoring one re-inserts every document and rebuilds every index, so it moves at MongoDB’s speed, not the array’s. I still want one. I just use each tool where it fits: snapshots when I need the cluster back fast, a dump when I need a copy that any mongod can read. And the dump goes on a system that isn’t holding the live data.
My Lab Setup
The cluster is the same rs-multi replica set from the last three posts, now with four members after the expansion onto a fourth array. None of the per-member volume layout from the earlier posts comes into play here, because mongodump and mongorestore work through mongod over the network and never touch the volumes underneath.
| Component | Name | Role |
|---|---|---|
| Replica set | rs-multi |
aen-mongo-05, -06, -07, and -04 on flasharray01 through flasharray04, one member per array |
| Dump host | aen-mongo-05 |
Runs mongodump and mongorestore, mounts the file system. It’s the primary, but the tools only run here: the dump’s reads go to a secondary |
| FlashBlade | flashblade01 |
Data VIP 192.168.20.54 |
| File system | aen-mongo-dumps |
NFS v4.1, mounted at /mnt/mongo-dumps |
| Export rule | 192.168.10.0/24 |
The subnet my MongoDB nodes live on |
| Tools | MongoDB Database Tools 100.15.0 | mongodump and mongorestore |
The demo database holds 4.83 GB of data in four collections, everything the earlier posts left behind. There’s orders, with 1,107,793 documents, and filler, with 2,240,000, the padding I wrote in the reseed post to roll the oplog past aen-mongo-06’s snapshot. The load collection (61,100 documents) and counters (100) are left over from the load tests.
I’m using Python again with py-pure-client 1.99.0, which ships a FlashBlade client right alongside the FlashArray one. connect_fb logs in to the FlashBlade, a FlashBlade//S200 running Purity//FB 4.8.7, with the same account the FlashArray scripts use:
def connect_fb(name="flashblade01"):
return flashblade.Client(f"{name}.fsa.lab", username=os.environ["FA_USERNAME"],
password=os.environ["FA_PASSWORD"], verify_ssl=False)
These are the names I use throughout:
NODE = "aen-mongo-05"
URI = ("mongodb://aen-mongo-05.fsa.lab:27017,aen-mongo-06.fsa.lab:27017,"
"aen-mongo-07.fsa.lab:27017,aen-mongo-04.fsa.lab:27017/?replicaSet=rs-multi")
TOOLS = "/path/to/mongodb-database-tools-100.15.0/bin" # where mongodump and mongorestore live on the node
FS = POLICY = "aen-mongo-dumps"
VIP = "192.168.20.54" # a FlashBlade data VIP
MNT = "/mnt/mongo-dumps"
DUMP = "dump02" # the dump directory and the snapshot suffix; this is my second dump into this file system
ssh() and mongosh() are the same small helpers as in the earlier posts. They run a shell command or a mongosh script on a node and return what it printed.
What You’ll Need
Here’s what I’m running, and what you’ll need to follow along:
- A FlashBlade: A FlashBlade//S200 running Purity//FB 4.8.7 in my lab, with a data VIP your MongoDB hosts can reach over NFS. The account needs to be able to create file systems, export policies, and file system snapshots.
- A MongoDB replica set: Any replica set will do, because the dump and the restore go through
mongod. Mine isrs-multi, MongoDB Enterprise 8.0.21 on RHEL 9.6. - MongoDB Database Tools on the dump host: Version 100.15.0 here, for
mongodumpandmongorestore, plus the host’s NFS client to mount the file system. - Room on the file system: My file system is 1 TiB, and this dump came out at 4.5 GiB. Size yours from your own
dataSize, plus room for however many dumps you keep before the snapshots age out. - A control machine: Python 3 with
py-pure-client1.99.0, SSH key access to the dump host as a user with passwordless sudo for the mount, and the FlashBlade credentials inFA_USERNAMEandFA_PASSWORD.
Step 1: Create the File System and Export
I create three things on the FlashBlade: an NFS export policy, one rule on that policy that lets my node subnet in, and a file system that uses the policy. Then I mount it on aen-mongo-05.
fb = connect_fb()
fb.post_nfs_export_policies(names=[POLICY], policy=NfsExportPolicyPost(enabled=True))
fb.post_nfs_export_policies_rules(policy_names=[POLICY],
rule=NfsExportPolicyRule(client="192.168.10.0/24", access="no-squash", permission="rw", security=["sys"]))
fb.post_file_systems(names=[FS], file_system=FileSystemPost(
provisioned=1 << 40, snapshot_directory_enabled=True,
nfs=Nfs(v4_1_enabled=True, export_policy=Reference(name=POLICY))))
ssh(NODE, f"sudo mkdir -p {MNT} && sudo mount -t nfs4 -o minorversion=1 {VIP}:/{FS} {MNT} && sudo chown packer {MNT}")
print(ssh(NODE, f"findmnt -no SOURCE,FSTYPE,SIZE {MNT}"))
A few notes on those choices:
- NFS v4.1 only: One node mounts this file system over NFS, so v4.1 is the only protocol I turn on.
no-squash: I mount as root and hand the directory to my user, so root on the client stays root. Check what your security team expects here before you copy this line.snapshot_directory_enabled=True: This is what exposes snapshots as a.snapshotdirectory on the mount. The whole restore in Step 4 depends on it.
Output:
192.168.20.54:/aen-mongo-dumps nfs4 1T
That one line is worth reading field by field. The source is the data VIP and the file system name, 192.168.20.54:/aen-mongo-dumps, so the node is talking to FlashBlade and not some other export. The type is nfs4, and the minorversion=1 in the mount command makes it 4.1, the only version the file system allows. The size is 1T.
Step 2: Dump from a Secondary
I run mongodump on the node with --readPreference=secondary, so a secondary serves the reads and the primary keeps doing its job. The output goes straight to the FlashBlade mount. I read the size of the demo database first, so there’s something to compare the dump against.
size = mongosh(NODE, 'print((db.getSiblingDB("demo").stats().dataSize / 1e9).toFixed(2) + " GB of data in the demo database")', uri=URI)
print(size)
t0 = time.time()
ssh(NODE, f"{TOOLS}/mongodump --uri='{URI}' --readPreference=secondary --numParallelCollections=4 --out {MNT}/{DUMP}")
secs = time.time() - t0
print(ssh(NODE, f"du -sh {MNT}/{DUMP}; ls {MNT}/{DUMP}/demo"))
print(f"mongodump took {secs:.0f} seconds")
Output:
4.83 GB of data in the demo database
4.5G /mnt/mongo-dumps/dump02
counters.bson
counters.metadata.json
filler.bson
filler.metadata.json
load.bson
load.metadata.json
orders.bson
orders.metadata.json
mongodump took 17 seconds
The dump is 4.5 GiB on FlashBlade, and it took 17 seconds. Don’t let the two sizes throw you, because they’re the same number. dataSize is the uncompressed size of the documents in bytes, and my script divides it by 10^9, so the 4.83 at the top is decimal gigabytes. du -h reports binary units, and 4.83 GB is 4.50 GiB. A dump is the documents written out as BSON, so it lands right at the dataSize. Each collection shows up as a pair of files: the .bson file with the documents, and a .metadata.json file with the collection options and index definitions. That metadata is how mongorestore knows which indexes to rebuild later.
--numParallelCollections=4 is the default for both tools. I set it so you can see the knob, because it’s the one to raise on a database with dozens of collections. Here demo has four, so all of them dump at once.
I ran this against an idle cluster. On a live replica set, add --oplog so the dump also captures the operations that happen while it runs, and replay them with --oplogReplay on the restore. Without it, each collection is read at a slightly different moment, and on a busy system that’s not a single point in time. --oplog needs a full dump of a replica set member, which is what this is, since I didn’t pass --db. The catch is on the restore side. mongorestore --oplogReplay fails if you add any option that narrows or renames what it restores: --db, --collection, --nsInclude, --nsExclude, --nsFrom, or --nsTo. A point-in-time restore brings back the whole dump under its original names, so point it at a separate scratch mongod. The rename into a scratch database in Step 4 works here because I’m not replaying an oplog.
Step 3: Snapshot the File System and Delete the Dump
A dump is a directory of files, and anyone with write access to the mount can delete it. A FlashBlade file system snapshot is what protects it. I take one, and then I delete the dump. Yes, on purpose.
fb = connect_fb()
fb.post_file_system_snapshots(source_names=[FS], file_system_snapshot=FileSystemSnapshotPost(suffix=DUMP))
print("Snapshot:", [s.name for s in fb.get_file_system_snapshots(filter=f"name='{FS}.{DUMP}'").items])
ssh(NODE, f"rm -rf {MNT}/{DUMP}")
print("After rm:", ssh(NODE, f"ls {MNT}"))
print("In the snapshot:", ssh(NODE, f"ls {MNT}/.snapshot/{FS}.{DUMP}/{DUMP}/demo"))
print(ssh(NODE, f"cd {MNT}/.snapshot/{FS}.{DUMP}/{DUMP} && ls && ls admin"))
Output:
Snapshot: ['aen-mongo-dumps.dump02']
After rm:
In the snapshot: counters.bson
counters.metadata.json
filler.bson
filler.metadata.json
load.bson
load.metadata.json
orders.bson
orders.metadata.json
admin
demo
prelude.json
system.version.bson
system.version.metadata.json
Look at the After rm: line. It’s empty, because the live file system has nothing left in it. The next line lists the same eight files from Step 2, read from .snapshot/aen-mongo-dumps.dump02/dump02/demo. The snapshot is named <file system>.<suffix>, and that’s also the name of its directory under .snapshot, so you can build the path from the name the API gives you back. The dump is gone from the file system and fully intact in the snapshot.
The last five lines are the top level of the dump in the snapshot, then what’s inside admin, and they show something the demo listing doesn’t: the dump has an admin database in it, not just demo. I didn’t pass --db, so mongodump copied every database it includes by default. It always skips local, and from config it keeps only the sharding metadata (chunks, shards, and friends), which a replica set doesn’t have. That leaves demo and admin.system.version. There’s also prelude.json, a small metadata file mongodump writes next to the data. Keep that admin directory in mind, because it comes back in the next step.
On deletion protection: This is a regular snapshot, and anyone with admin rights on the FlashBlade can destroy a regular snapshot. If the point of this copy is surviving someone with credentials and bad intentions, SafeMode is the feature to put in front of your storage team.
Step 4: Restore from the Snapshot into a Scratch Database
Now the part I like. I don’t copy the dump back. mongorestore reads directly from the read-only snapshot directory, and I rename the namespace with --nsFrom and --nsTo so everything lands in a scratch database called demo_restored. That way I can check the restore against the live data without touching it.
t0 = time.time()
log = ssh(NODE, f"{TOOLS}/mongorestore --uri='{URI}' --numParallelCollections=4 --nsFrom='demo.*' --nsTo='demo_restored.*' "
f"--dir {MNT}/.snapshot/{FS}.{DUMP}/{DUMP} 2>&1 | grep -E 'admin|restored successfully' | cut -c30-")
print(f"mongorestore took {time.time() - t0:.0f} seconds")
print(log)
for db in ("demo", "demo_restored"):
print(db, mongosh(NODE, f'print(db.getSiblingDB("{db}").orders.countDocuments())', uri=URI))
mongosh(NODE, 'db.getSiblingDB("demo_restored").dropDatabase()', uri=URI)
I keep two kinds of lines from mongorestore’s log: anything that mentions admin, and its closing summary. The cut trims the timestamp off the front.
Output:
mongorestore took 63 seconds
3408993 document(s) restored successfully. 0 document(s) failed to restore.
demo 1107793
demo_restored 1107793
The restore took 63 seconds, and the orders collection has 1,107,793 documents in both demo and demo_restored. Look at the total in the summary line too. 3,408,993 is exactly the four demo collections added up: 1,107,793 orders, 2,240,000 filler, 61,100 load, and 100 counters. There’s no line about admin, even though I was grepping for one. The --dir path is the snapshot path from Step 3, so nothing moved back onto the live file system first. The live demo database was never touched, and the last line of the script drops demo_restored once the counts are in.
Compare the 63 seconds with the 17 it took to dump. That’s the cost I mentioned at the top. The dump only reads documents and writes files. The restore has to insert every document through the primary, replicate it to the other three members, and rebuild the indexes from the .metadata.json files. That gap grows with your data, so build it into your recovery time.
Now back to that admin directory. The 3,408,993 total says only demo came back this time, but the dump holds more than demo, and I don’t want the scope of a restore to depend on what mongorestore happens to skip. So add --nsInclude='demo.*' when you only want one database back. Then the restore’s scope is something you set, not something you check after the fact.
Why This Matters
- The copy lives on a different system: The data is on FlashArray and the dump is on FlashBlade. A problem with the arrays holding the live data doesn’t reach the copy you’d use to recover from it.
- A deletion becomes survivable: I removed the dump with
rm -rf, and the restore didn’t care. The snapshot is what turns a directory of files into a backup. - There’s no copy-back step:
mongorestorereads from.snapshotdirectly, so recovering a dump doesn’t start with copying four and a half gigabytes back onto the file system and waiting. - You can prove a backup is good without touching production: Restoring into a scratch name with
--nsFromand--nsTolets you check counts against the live data, then drop the copy. - This is the restore that grows with your data: A logical restore re-inserts every document and rebuilds every index. An earlier run of this same test, with 2.13 GB in
demo, restored in 24 seconds. This one, with 4.83 GB, took 63. Compare that with the protection group restores in the earlier posts, where the array’s part takes the same time at 10 GB or 10 TB. That’s why the series uses both: snapshots for fast recovery, and dumps on FlashBlade for a portable copy somewhere else.
Wrapping Up
FlashBlade isn’t where MongoDB lives in this series, but it’s a natural home for the copies you want off the cluster’s storage. A dump from a secondary, a file system snapshot, and a restore read straight from that snapshot took a handful of API calls and two MongoDB tools. In my lab the dump took 17 seconds, and the restore brought back all 3,408,993 documents, including all 1,107,793 orders, in 63, after I’d deleted the dump itself.
Two things to keep in your head when you do this yourself. First, on a live replica set, add --oplog to the dump and --oplogReplay to the restore, or the dump isn’t a single point in time. --oplogReplay restores the whole dump under its own names, so it goes to a scratch mongod, not a renamed database. Second, a dump without --db includes admin along with your data, so when you restore one database from a dump without replaying the oplog, add --nsInclude and make the scope explicit. FlashBlade also speaks S3, and mongoexport gives you JSON or CSV instead of BSON, but those are a post for another day. Before you build on this, read MongoDB’s documentation for mongodump and mongorestore, especially the sections on --oplog and the namespace options. Then get out in your lab, point it at a test replica set, and let me know how it works in your environment.
MongoDB on Everpure Series
- Why Run MongoDB on Everpure FlashArray?
- Snapshot Recovery of a MongoDB Replica Set on a Single FlashArray
- Replacing the MongoDB Replica Set Initial Sync with FlashArray Snapshots
- When You Need fsyncLock: Restoring a MongoDB Replica Set from Local Snapshots on Every FlashArray
- Expanding a MongoDB Replica Set with Everpure Fusion
- Where FlashBlade Fits When MongoDB Lives on FlashArray (This Post)
Related: App-Consistent MongoDB Snapshots Across Multiple Pure Storage FlashArrays, my earlier post on snapshotting a sharded MongoDB cluster across two arrays.