Snapshot Recovery of a MongoDB Replica Set on a Single FlashArray

Page content

I’ve been running MongoDB on Everpure FlashArray in my lab, and the first thing I wanted to prove is the simplest one. Can I snapshot an entire replica set while it’s taking writes, drop a database, and get everything back just by overwriting volumes?

In this post, I’m going to show you exactly that: a three-member replica set, one mongod per node, every member on volumes from the same FlashArray, four writers and an updater hammering away during the snapshot, and no fsyncLock. If you’ve read my intro to this series, this is the first hands-on post. If protection groups, the oplog, or crash consistency are new to you, the terms list in the intro defines them in plain words. Let’s go.

The Problem We’re Solving

A replica set protects you from a server going down. It doesn’t protect you from yourself. If someone drops a database, that dropDatabase replicates to every member, and now you have three perfectly synchronized copies of nothing. What you need is a point in time you can go back to.

The usual answers are a logical dump, a byte by byte, document by document copy, which gets slower to take and slower to restore as the data grows, or a snapshot of the files underneath mongod. Snapshots are fast, but MongoDB’s data files are only safe to copy if you get a consistent image and you still have to copy them somewhere else for safety. The textbook way to get a consistent image is to lock the database, snapshot, and unlock. That works, and I used it in my multi-array snapshot post, but writes block for as long as the lock is held.

You can skip the lock when each member’s volumes are snapshotted together. That’s what I’m testing here.

My Lab Setup

Three VMs, each running one mongod. Each VM gets four volumes from the same FlashArray, flasharray01, presented as physical RDMs and formatted XFS:

Mount Volume What lives there In the protection group
/u01/data aen-mongo-single-0N-data Data files and indexes Yes
/u01/journal aen-mongo-single-0N-journal WiredTiger journal Yes
/u01/oplog aen-mongo-single-0N-oplog The local database, which holds the oplog Yes
/u01/log aen-mongo-single-0N-log mongod.log No
   aen-mongo-01            aen-mongo-02            aen-mongo-03
   mongod (PRIMARY)        mongod (SECONDARY)      mongod (SECONDARY)
   data journal oplog log  data journal oplog log  data journal oplog log
        │                       │                       │
        └───────────────────────┼───────────────────────┘
                                │
                          flasharray01
   
     protection group aen-mongo-pg-single: every data, journal, and oplog volume (9)

     log volumes (3): outside the group

I split these out for management and separation of concerns, not for performance. On a FlashArray these volumes all land on the same flash, so a separate journal or oplog volume isn’t a performance lever on the array side. What you get is a layout a lot of MongoDB teams already run, where each piece can be sized, monitored, and protected on its own. It also lets me make a deliberate choice about the log, which I’ll come back to in Step 7.

How MongoDB Gets Pointed at Each Volume

MongoDB doesn’t let you pick a path for most of these. There’s one dbPath, and WiredTiger always puts the journal in dbPath/journal. With --directoryperdb, each database gets its own directory, dbPath/<database>, and the oplog lives in the local database. So I mount the volumes at /u01/journal and /u01/oplog and make dbPath/journal and dbPath/local symlinks to them. The log is the easy one, because --logpath takes any path.

The snippets in this series lean on a handful of small helpers, so here’s what each one does:

  • ssh(node, cmd): Runs a shell command on a node over SSH and returns its output. It retries if the connection drops, which my lab network does now and then when a lot of sessions open at once.
  • mongosh(node, js, uri=None): Runs a bit of JavaScript through mongosh on a node and returns what it printed. Without uri it talks to the local mongod; with uri it connects wherever that connection string points.
  • background(node, js, log): Starts a mongosh script on a node with nohup, against the replica set, so it keeps running after the SSH session ends. That’s how the load writers in Step 5 run.
  • count(coll="load"): Counts the documents in one demo collection through the replica set connection string.
  • stop_members(): Shuts down mongod cleanly on each node with mongod --shutdown.
  • start_members(): Creates the dbPath/journal and dbPath/local symlinks if they aren’t there yet and starts mongod on each node, node 01 first, with the command below.
  • wait_for_primary(): Polls db.hello() on aen-mongo-01 every few seconds until it’s a writable primary.

And the names: NODES is the list of three VMs, DBPATH is /u01/data/mongod, BIN is where the MongoDB binaries live, and URI is the replica set connection string, mongodb://aen-mongo-01,aen-mongo-02,aen-mongo-03/?replicaSet=rs-single with each host on port 27017.

ssh(n, f"sudo runuser -u mongod -- mkdir -p {DBPATH} && "
       f"( [ -L {DBPATH}/journal ] || sudo runuser -u mongod -- ln -s /u01/journal {DBPATH}/journal ) && "
       f"( [ -L {DBPATH}/local ] || sudo runuser -u mongod -- ln -s /u01/oplog {DBPATH}/local )")
ssh(n, f"sudo runuser -u mongod -- {BIN}/mongod --replSet rs-single --bind_ip_all --dbpath {DBPATH} --directoryperdb "
       f"--logpath /u01/log/mongod.log --logappend --wiredTigerCacheSizeGB 2 --fork")

Two things to know here. --directoryperdb has to be set when the data directory is first created, so plan for it up front. And the symlinks themselves live on the data volume, so restoring the data volume brings them back along with everything else. That’s what you want, since they point at the same mount paths on every restore.

The replica set is called rs-single. I’m using Python with the py-pure-client package, because it runs anywhere your MongoDB hosts do. Install it, and hand the script your Fusion gateway array’s REST endpoint and credentials through environment variables:

pip install py-pure-client
export FA_ENDPOINT=<Fusion gateway array host name, no https://> FA_USERNAME=<user> FA_PASSWORD=<password>

What You’ll Need

Here’s what I’m running, and what you’ll need to follow along:

  • A FlashArray: All twelve volumes for all three nodes live on one array, running Purity//FA 6.12 in my lab.
  • An Everpure Fusion fleet: The array is a member of a fleet, and I connect to the fleet’s gateway array, which is why every call passes context_names. That argument tells the gateway which array the call is for. If your array isn’t in a fleet, connect to it directly; context_names only matters when one connection has to reach several arrays.
  • Three Linux hosts: RHEL 9.6 in my lab, one mongod on each. Each host gets four volumes: data at 1 TiB, and journal, oplog, and log at 100 GiB each. Format them XFS, mount them at /u01/data, /u01/journal, /u01/oplog, and /u01/log, and add them to fstab. My hosts are VMware VMs with the volumes presented as physical RDMs. On bare metal, connect the volumes to your host or host group the way you normally would.
  • A protection group: One group on the array that holds every member’s data, journal, and oplog volumes, and none of the log volumes.
  • MongoDB 8.0 and mongosh: I’m on MongoDB Enterprise 8.0.21 and mongosh 2.8.1. Each mongod starts with --directoryperdb, and dbPath/journal and dbPath/local have to be symlinked to the journal and oplog mounts before the first start, as shown above.
  • A control machine: Python 3 with py-pure-client (1.99.0 here), and SSH key access to each host as a user with passwordless sudo. The scripts start mongod as the mongod user with runuser, and read each device’s SCSI serial with /usr/lib/udev/scsi_id.
  • Array credentials: An account that can take protection group snapshots and overwrite volumes, passed in through FA_ENDPOINT, FA_USERNAME, and FA_PASSWORD.

If you’re starting from scratch, the protection group is two calls. Create it, then add each member’s data, journal, and oplog volumes:

fa.post_protection_groups(names=["aen-mongo-pg-single"], context_names=["flasharray01"])
fa.post_protection_groups_volumes(group_names=["aen-mongo-pg-single"], context_names=["flasharray01"],
    member_names=[f"aen-mongo-single-0{i}-{role}" for i in (1, 2, 3) for role in ("data", "journal", "oplog")])

Why This Works Without fsyncLock

A protection group snapshot is atomic on the array. Every volume in the group is captured at the same instant, and from MongoDB’s point of view that’s exactly what the disks would look like if the power cut out at that moment. WiredTiger is built to recover from that. On startup it opens the data files at its last checkpoint and replays the journal, then mongod replays its own oplog, and the member is back to a consistent state. I walk through those stages in detail in the next post.

The split layout makes this a tougher test. MongoDB’s documentation on backups with file system snapshots is blunt about it: “If your mongod instance has the journal files on a separate volume, you must flush all writes to disk and lock the database to prevent writes during the backup process.” The reason is timing. Snapshot two volumes one after the other and the journal no longer matches the data files it describes. A protection group doesn’t have that problem. It captures each member’s data, journal, and oplog volumes at the same instant, so the journal still lines up with the data files, and there’s nothing to lock.

The thing that matters the most is per replica set member: every volume a mongod writes its data, journal, and oplog to has to be in the same protection group. I went one step further and put all three members in one group, which gets me the whole replica set in a single snapshot call. That part is a convenience.

Where you do need to quiesce is a sharded cluster. MongoDB’s procedure for those has you stop the balancer and lock the cluster, because chunks can migrate during the backup, and the shards and config servers have to line up with each other. That’s what my multi-array post covered, across arrays that don’t share one atomic snapshot.

Let’s get started with using snapshots for replica sets.

Step 1: Find the Volumes Behind Each Node

I don’t want to hardcode which volume belongs to which node. I connect once to the Everpure Fusion gateway, read the SCSI serial of the device behind each mount on each node, and search the fleet for the volume with that serial. Every call that targets a particular array takes context_names, which is how Fusion routes it to that array even though I only connected to one. In the code, fa is that gateway connection, arrays is the list of arrays in the fleet, and MOUNTS maps each role to its mount path. Doing this, you won’t have to maintain a configuration file of which mongodb volumes are where in your Everpure environment. Everybody say “thanks Fusion”….“Thanks Fusion!!!”.

def volume_serial(node, mount):
    dev = ssh(node, f"findmnt -no SOURCE {mount}")
    wwn = ssh(node, f"sudo /usr/lib/udev/scsi_id --whitelisted --device={dev}")
    return wwn.removeprefix("3624a9370").upper()

def find_volume(fa, arrays, serial):
    for array in arrays:
        r = fa.get_volumes(filter=f"serial='{serial}'", context_names=[array])
        if isinstance(r, flasharray.ErrorResponse):
            continue
        items = list(r.items)
        if items:
            return items[0].name, array
    raise LookupError(f"no volume with serial {serial}")

for n in NODES:
    for role, path in MOUNTS.items():
        layout[(n, role)] = find_volume(fa, arrays, volume_serial(n, path))
        print(f"{n:14} {path:13} {layout[(n, role)][0]:28} {layout[(n, role)][1]}")

Output:

aen-mongo-01   /u01/data     aen-mongo-single-01-data     flasharray01
aen-mongo-01   /u01/journal  aen-mongo-single-01-journal  flasharray01
aen-mongo-01   /u01/oplog    aen-mongo-single-01-oplog    flasharray01
aen-mongo-01   /u01/log      aen-mongo-single-01-log      flasharray01
aen-mongo-02   /u01/data     aen-mongo-single-02-data     flasharray01
aen-mongo-02   /u01/journal  aen-mongo-single-02-journal  flasharray01
aen-mongo-02   /u01/oplog    aen-mongo-single-02-oplog    flasharray01
aen-mongo-02   /u01/log      aen-mongo-single-02-log      flasharray01
aen-mongo-03   /u01/data     aen-mongo-single-03-data     flasharray01
aen-mongo-03   /u01/journal  aen-mongo-single-03-journal  flasharray01
aen-mongo-03   /u01/oplog    aen-mongo-single-03-oplog    flasharray01
aen-mongo-03   /u01/log      aen-mongo-single-03-log      flasharray01

All twelve volumes resolve to flasharray01, which is the whole premise of this post, so the script exits if they span more than one array. scsi_id prints the device’s WWN with a leading 3, and every Pure volume’s WWN starts with 624a9370, so stripping 3624a9370 leaves exactly the volume serial the array reports. One thing cost me a few minutes here. When you pass context_names, find the volume with filter= on the serial instead of looking it up with names=. The lookup by name returns an error.

Step 2: Check the Protection Group

Next, the script checks that every data, journal, and oplog volume is in aen-mongo-pg-single, and that no log volume is. It stops if either check fails. PG is aen-mongo-pg-single, and array is the one array all twelve volumes resolved to.

members = {m.member.name for m in fa.get_protection_groups_volumes(group_names=[PG], context_names=[array]).items}
for (n, role), (vol, _) in sorted(layout.items()):
    print(f"{vol:28} {'in ' + PG if vol in members else 'not in the protection group'}")

Output:

aen-mongo-single-01-data     in aen-mongo-pg-single
aen-mongo-single-01-journal  in aen-mongo-pg-single
aen-mongo-single-01-log      not in the protection group
aen-mongo-single-01-oplog    in aen-mongo-pg-single
aen-mongo-single-02-data     in aen-mongo-pg-single
aen-mongo-single-02-journal  in aen-mongo-pg-single
aen-mongo-single-02-log      not in the protection group
aen-mongo-single-02-oplog    in aen-mongo-pg-single
aen-mongo-single-03-data     in aen-mongo-pg-single
aen-mongo-single-03-journal  in aen-mongo-pg-single
aen-mongo-single-03-log      not in the protection group
aen-mongo-single-03-oplog    in aen-mongo-pg-single

Nine volumes in the group, three out. That’s the whole design in one listing. I like having this check up front, because the failure it catches is quiet: if a member’s oplog volume never made it into the group, or someone added a log volume to it later, the snapshot would still succeed and you’d only find out at restore time. Here the script stops before anything changes. And given how dynamic a mongo cluster can be in terms of nodes and replica topology it’s important to have a safety net like this and it’s equally important to test your restores often so configuration errors can be addressed quickly.

Step 3: Build the Replica Set

I start one mongod on each node and initiate the replica set from the first one. I give aen-mongo-01 a higher priority so it’s the primary, which means the load and the counts in the next steps all run through it. HOSTS holds each node’s host:27017 address.

mongosh(NODES[0], f"""rs.initiate({{ _id: 'rs-single', members: [
  {{ _id: 0, host: '{HOSTS[0]}', priority: 2 }},
  {{ _id: 1, host: '{HOSTS[1]}' }},
  {{ _id: 2, host: '{HOSTS[2]}' }} ] }})""")

Output:

aen-mongo-01.fsa.lab:27017  PRIMARY
aen-mongo-02.fsa.lab:27017  SECONDARY
aen-mongo-03.fsa.lab:27017  SECONDARY

aen-mongo-01 is PRIMARY because of that priority: 2. The other two default to priority 1, so as long as node 01 is healthy and caught up, it’s the member the set keeps electing. That matters again in Step 7, when all three come back from the snapshot at once.

Step 4: Load Some Data and Check Where It Landed

Next, I load a million documents into demo.orders, each with 200 characters of padding. Then I check that MongoDB actually put each piece where I meant it to go, because it’s easy to get a symlink wrong.

print(f"Documents loaded: {count('orders')}")  # documents in a demo collection, load by default
print(ssh(NODES[0], f"ls -l {DBPATH} | grep -E ' (journal|local) ' | awk '{{print $9, $10, $11}}'"))
# then lists what's on each mount, shown in the last four lines below

Output:

Documents loaded: 1000000
journal -> /u01/journal
local -> /u01/oplog
/u01/journal: WiredTigerLog.0000000001 WiredTigerPreplog.0000000001
/u01/oplog:   17 WiredTiger files
/u01/log:     mongod.log
/u01/data:    admin config demo

That’s what I wanted to see. The two symlinks point where they should. The WiredTigerLog and WiredTigerPreplog files, which are the WiredTiger journal, are on /u01/journal. The local database, oplog included, is 17 WiredTiger files on /u01/oplog. The log is alone on /u01/log, and the admin, config, and demo databases are on /u01/data.

Step 5: Snapshot Under Load

Now the fun part. A single writer makes for an easy test, so I put the replica set under real load instead. Four writers insert documents as fast as they can, and each one numbers its own documents with an increasing sequence. A fifth process runs a steady stream of updates against 100 counter documents. A sampler reads serverStatus on the primary once a second, so I can see what the snapshot does to throughput. background() starts a mongosh script on a node with nohup, so it keeps running while the rest of the script moves on.

for w in range(WRITERS):
    # run this mongosh script on the node with nohup
    background(NODES[0], f"""const c = db.getSiblingDB('demo').load; const pad = 'p'.repeat(200); let s = 0;
while (true) {{ c.insertOne({{ _id: 'w{w}-' + s, w: {w}, seq: s, pad: pad }}); s++;
  if (s % 100 === 0 && require('fs').existsSync('/tmp/stop-load')) break; }}""", f"/tmp/writer{w}.log")

After 20 seconds of load, I take the snapshot. I read the document count right before and right after the call, which gives me a window the restore has to land in. count() returns countDocuments() on demo.load through the replica set connection string, URI.

before = count()  # documents in a demo collection, load by default
snap = fa.post_protection_group_snapshots(source_names=[PG], context_names=[array],
                                          protection_group_snapshot=ProtectionGroupSnapshotPost(suffix="load01"))
after = count()   # documents in a demo collection, load by default

Output:

Snapshot: aen-mongo-pg-single.load01 (9 volumes)
Count before snapshot call: 51128
Count after snapshot call:  58444
ops per second on the primary (inserts + updates), one sample per second
  t -5.5s   2236 inserts    553 updates
  t -4.4s   2225 inserts    553 updates
  t -3.4s   2154 inserts    520 updates
  t -2.4s   2302 inserts    545 updates
  t -1.4s   2188 inserts    521 updates
  t -0.4s   2229 inserts    531 updates
  t +0.6s   2104 inserts    507 updates   <- snapshot
  t +1.6s   2196 inserts    515 updates   <- snapshot
  t +2.6s   2095 inserts    497 updates
  t +3.6s   2225 inserts    532 updates
  t +4.6s   2187 inserts    540 updates
  t +5.6s   2180 inserts    537 updates
  t +6.6s   2157 inserts    523 updates
average before: 2705/s, during: 2661/s, after: 2778/s

Look at the two samples marked <- snapshot, the seconds that overlap the snapshot call. Inserts stay above 2,100 a second and updates above 500, right in line with the seconds around them. The average line says it in one place: 2,705 operations a second before, 2,661 during, and 2,778 after. There’s no stall. One call captured nine volumes across three members while four writers and an updater kept going, and across ten more runs of this test the call itself took about a second every time (1,004 to 1,078 ms).

The load count moved by 7,316 documents between my two reads, so a correct restore has to land somewhere between 51,128 and 58,444.

Step 6: Break It

Fifteen seconds after the snapshot, I stop the load and drop the database. Generally a resume generating event, but you’ll be amazed by how fast you can recover from a snapshot…well after that funny feeling in your stomach goes away because you just dropped a database in prod. After that, you’ll feel better. I promise.

print(mongosh(NODES[0], 'printjson(db.getSiblingDB("demo").dropDatabase())', uri=URI))

Output:

{
  ok: 1,
  dropped: 'demo'
}
Documents after drop: 0

ok: 1 and dropped: 'demo', and the count through the replica set connection is 0. The drop replicates to all three members, so every copy is gone, the million orders documents included. This is the case where replicas don’t save you, because they faithfully replicated the mistake.

Step 7: Restore by Overwriting the Volumes

To restore, I stop mongod on every member, unmount data, journal, and oplog, and overwrite those nine volumes from the snapshot. The log volume stays mounted the whole time. It isn’t in the protection group, so the restore never touches it.

It’s stop_members(), start_members(), and wait_for_primary() from the helper list, wrapped around the unmounts, the overwrites, and the mounts:

stop_members()      # mongod --shutdown on each node
for n in NODES:
    ssh(n, "sudo umount /u01/data /u01/journal /u01/oplog")
for n in NODES:
    for role in ["data", "journal", "oplog"]:
        vol = layout[(n, role)][0]
        fa.post_volumes(names=[vol], context_names=[array], overwrite=True,
                        volume=VolumePost(source=Reference(name=f"{PG}.load01.{vol}")))
for n in NODES:
    ssh(n, "sudo mount /u01/data && sudo mount /u01/journal && sudo mount /u01/oplog")
start_members()     # symlinks if needed, then mongod on each node
wait_for_primary()  # wait until aen-mongo-01 is a writable primary

Output:

Overwrote aen-mongo-single-01-data
Overwrote aen-mongo-single-01-journal
Overwrote aen-mongo-single-01-oplog
Overwrote aen-mongo-single-02-data
Overwrote aen-mongo-single-02-journal
Overwrote aen-mongo-single-02-oplog
Overwrote aen-mongo-single-03-data
Overwrote aen-mongo-single-03-journal
Overwrote aen-mongo-single-03-oplog

Restore finished in 77 seconds (includes stopping and starting mongod and the election)

Those Overwrote lines only say the calls were made. post_volumes returns an error response instead of raising one, so check what it returns in your own code. My repeat-run version exits on the first ErrorResponse. Here, the counts in Step 8 are the real proof. The snapshot name has the form <protection group>.<suffix>.<volume>, and that’s how the source reference finds the right copy of each volume. The overwrite replaces the contents of the existing volume, so the device, the serial, and the host connection all stay the same. I don’t touch the VMs at all.

The 77 seconds covers the whole sequence: stopping mongod on all three members, unmounting, overwriting nine volumes, remounting, starting mongod, and waiting for the election. The array’s part of that is nine metadata calls. The rest is SSH round trips to each node, mongod shutting down, WiredTiger and replication recovery on startup, and the election.

Step 8: Verify

Let’s see if our restore worked. So to verify, for each writer, the number of documents has to equal its highest sequence number plus one. A gap would mean the snapshot caught some writes and missed earlier ones.

gaps = mongosh(NODES[0], """db.getSiblingDB('demo').load.aggregate([{ $group: { _id: '$w', n: { $sum: 1 }, max: { $max: '$seq' } } },
  { $sort: { _id: 1 } }]).forEach(r => print('writer ' + r._id + ': ' + r.n + ' documents, highest seq ' + r.max +
  (r.n === r.max + 1 ? ', no gaps' : ', GAPS')))""", uri=URI)

Output:

Load documents after restore: 53838 (expected between 51128 and 58444)
Orders after restore: 1000000
writer 0: 14198 documents, highest seq 14197, no gaps
writer 1: 13661 documents, highest seq 13660, no gaps
writer 2: 13042 documents, highest seq 13041, no gaps
writer 3: 12937 documents, highest seq 12936, no gaps
...output omitted...
PASS

We got back 53,838 load documents, inside the window, plus all 1,000,000 orders. Add up the four writers and you get exactly 53,838, and each one’s count is its highest seq plus one, so every sequence is unbroken from zero. That’s what a crash-consistent snapshot taken during live writes should give you, and it’s a much stronger signal than a count alone.

One run is a demo, so I ran Steps 5 through 8 ten more times against the same replica set, with a new snapshot each time. A run only passed if the restored count landed in its window, no writer had a gap, all 1,000,000 orders came back, the demo dbHash matched on all three members, the set was still PRIMARY/SECONDARY/SECONDARY 20 seconds later, and no member’s log showed a rollback or Too stale since the run started. All ten passed, with restores between 74 and 93 seconds.

And here’s the log volume after all of that:

sudo head -1 /u01/log/mongod.log | cut -c1-40
echo "mongod starts recorded in this log: $(sudo grep -c '"msg":"MongoDB starting"' /u01/log/mongod.log)"
echo "dropDatabase recorded: $(sudo grep -c dropDatabase /u01/log/mongod.log)"

Output:

{"t":{"$date":"2026-11-16T22:06:30.785+0
mongod starts recorded in this log: 2
dropDatabase recorded: 9

The oldest line, at 22:06:30, is from the first time mongod started, before the load and before the snapshot. The log records both starts, the original and the one after the restore, and it still has the record of the drop, nine lines that mention dropDatabase from the one drop in Step 6. The database went back in time. The log of how it got there didn’t.

Why This Matters

  1. No pause in writes: The replica set did 2,705 operations a second before the snapshot call and 2,661 during it. There’s no lock to hold and no window where writes block.
  2. The journal can live on its own volume: MongoDB’s docs say a journal on a separate volume means flushing and locking for a coherent backup. A protection group captures data, journal, and oplog at the same instant, so the lock isn’t needed.
  3. One call covers every member: The snapshot captured nine volumes across three members at the same instant. The members don’t have to be at the identical oplog position, because secondaries trail the primary a little and the snapshot freezes each one wherever it was. That’s fine, because MongoDB’s election rules say a member can’t become primary unless it has the most recent optime of any member it can see, so whoever’s behind just catches up from the oplog. In the ten repeat runs, no member logged a rollback and the dbHash matched every time.
  4. Restore time doesn’t depend on the size of the data: Overwriting a volume from a snapshot is a metadata operation on the array, so putting back nine volumes takes the same time whether the database is 10 GB or 10 TB. The 77 seconds here went to the SSH calls, unmounting and remounting, and MongoDB stopping, recovering, and electing a primary, wrapped around nine metadata calls to the array. A restore that copies the data back, from a backup file or over the network, grows with every byte you have.

Wrapping Up

The bottom line is that a protection group snapshot on one array gives you a crash-consistent copy of a whole replica set while it’s under load, with the data, journal, and oplog on separate volumes and no fsyncLock. Restoring it is a volume overwrite and a restart. In my lab that took 77 seconds start to finish, the count landed inside the window, every writer’s sequence came back without a gap, and ten more runs all passed.

There are two things to keep in your head when you do this yourself. First, the rule is per member: each mongod’s data, journal, and oplog volumes go in the same protection group, and anything you don’t want rewound, like the log, stays out. Second, this is a single-array design. Even with 99.9999% uptime you have to plan on an array loss in mission critical systems. Lose the array and you lose all three members, and a snapshot on the same array is no substitute for a copy somewhere else. You’ve got three good options, and I’ll cover each one later in this series:

  • Spread your replicas across multiple FlashArrays, so losing an array costs you one member and not the whole set. The reseed post runs a replica set with each member on its own array, and the Fusion post adds a new member on a different array.
  • Replicate the protection group’s snapshots to another FlashArray.
  • Run logical backups onto FlashBlade.

In this post, I restored all three members at once, which is the easy case. Restoring just one member depends on the oplog window, and that’s the next post.

Before you rely on any of this, read MongoDB’s documentation on backups with file system snapshots. Then get out in your lab and try it. Let me know how it works in your environment.

MongoDB on Everpure Series

  1. Why Run MongoDB on Everpure FlashArray?
  2. Snapshot Recovery of a MongoDB Replica Set on a Single FlashArray (This Post)
  3. Replacing the MongoDB Replica Set Initial Sync with FlashArray Snapshots
  4. When You Need fsyncLock: Restoring a MongoDB Replica Set from Local Snapshots on Every FlashArray
  5. Expanding a MongoDB Replica Set with Everpure Fusion
  6. Where FlashBlade Fits When MongoDB Lives on FlashArray
  7. A Tag-Based Snapshot Catalog for a MongoDB Replica Set Across FlashArrays with Everpure Fusion

Related: App-Consistent MongoDB Snapshots Across Multiple Pure Storage FlashArrays, my earlier post on snapshotting a sharded MongoDB cluster across two arrays.