Replacing the MongoDB Replica Set Initial Sync with FlashArray Snapshots

Page content

In my last post, I restored every member of a replica set from one snapshot on the same array. In real life, you usually lose one member: a host dies, someone rebuilds the wrong VM, a member’s data gets damaged. And each member of a replica set is most likely on a different array. MongoDB rebuilds a member from nothing with an initial sync, the job every DBA dreads, and I wanted to see how much of that job I could hand off to the storage tier.

In this post, I’m going to show you how to rebuild a member without an initial sync. First from its own snapshot, kept recent enough to work. Then from a replicated snapshot of the primary, taken under load with no fsyncLock, and that same snapshot rebuilds every secondary in the set. Let’s go.

The Problem We’re Solving

A member that comes back while its last operation is still in another member’s oplog replays the difference and rejoins. That’s incremental and cheap, and MongoDB handles it on its own.

The trouble starts when that operation has rolled out of every oplog, or when the member is brand new or sits on a different array. The DBA’s normal answer is an initial sync: copy every document over the network and rebuild every index. That’s a size-of-data operation, and the member it syncs from carries the read load the whole time. Within a site and with a smaller data set, this might not be too challenging but between sites or where your data set is very large, this can really be painful.

The array can move the data for you, off the database hosts. After the first transfer, which is data reduced, it only sends the blocks that changed since the last replicated snapshot. And that’s the special sauce. If you’re working with a large data set, most of your data is already on the target array from the last replicated snapshot. You’re only resyncing because a replica lost track of the current state. FlashArray doesn’t send the same blocks twice between two arrays, just the ones that haven’t been replicated yet. That turns a resync from a size-of-data operation into a “what’s changed since the replica went offline” operation, dramatically reducing the resync operation’s time.

A replicated snapshot replaces the initial sync, and once the member is online, the oplog does the last bit of catching up, the way it always does.

My Lab Setup

If oplog windows or protection group targets are new to you, the terms list in the intro post covers both sides of the house.

This is a different replica set from the last post, rs-multi, with one mongod on each of three nodes. Each node’s volumes live on that node’s own FlashArray, so the replica set spans three arrays (flasharray01, flasharray02, and flasharray03).

   aen-mongo-05              aen-mongo-06              aen-mongo-07
   mongod (PRIMARY)          mongod (SECONDARY)        mongod (SECONDARY)
   data journal oplog log    data journal oplog log    data journal oplog log
        │                         │                         │
   flasharray01              flasharray02              flasharray03
   aen-mongo-pg-multi        aen-mongo-pg-multi        aen-mongo-pg-multi
        │                         ▲                         ▲
        ├──── async replication ──┘                         │
        └──── async replication ────────────────────────────┘

   each group holds that member's data, journal, and oplog, never its log
   only the primary's group, on flasharray01, has replication targets

Each member uses the same split volume layout as the last post. The log stays out of the protection group, so a restore rewinds data, journal, and oplog but never the log:

Mount Volume In aen-mongo-pg-multi
/u01/data aen-mongo-multi-0N-data Yes
/u01/journal aen-mongo-multi-0N-journal Yes
/u01/oplog aen-mongo-multi-0N-oplog Yes
/u01/log aen-mongo-multi-0N-log No

aen-mongo-05 is the primary and my seed source. Its protection group on flasharray01 replicates to flasharray02, aen-mongo-06’s array, over async replication. Later I add flasharray03 as a second target to rebuild aen-mongo-07 too.

What You’ll Need

Here’s what you’ll need to follow along:

  • Three FlashArrays: One member per array. In my lab, flasharray01 runs Purity//FA 6.12.4 and the other two run 6.10.7.
  • An Everpure Fusion fleet: All three arrays joined to one fleet. I connect to the fleet’s gateway array and route every call with context_names.
  • A replication connection from the seed member’s array to the target member’s array: That’s an async connection from flasharray01 to flasharray02 here, plus one to flasharray03 to rebuild every secondary.
  • A protection group on each array: Each holds that member’s data, journal, and oplog volumes, not its log. The one on flasharray01 has flasharray02 as a replication target.
  • Three Linux hosts with the four volumes from the last post: RHEL 9.6, MongoDB Enterprise 8.0.21, and mongosh 2.8.1 in my lab. For the demo I use a small oplog, --oplogSize 990, so the oplog window is easy to hit.
  • A control machine: Python 3 with py-pure-client 1.99.0, SSH key access to each host as a user with passwordless sudo, and array credentials in FA_ENDPOINT, FA_USERNAME, and FA_PASSWORD.

Step 1: Build the Replica Set

First I build the replica set with aen-mongo-05 at priority 2, load a million documents, and take a snapshot of aen-mongo-06’s protection group. ssh and mongosh run a command or a bit of JavaScript on a node, fa is my client on the fleet’s gateway array, and volume_serial and find_volume are the same serial lookups as the last post. The rest are in the comments.

# direct(node): a connection string to one member with directConnection=true
# stop(node): mongod --shutdown    members(): each member's name and state from rs.status()
# BIN: the MongoDB 8.0 bin directory    DOMAIN: my lab's DNS domain
NODES = ["aen-mongo-05", "aen-mongo-06", "aen-mongo-07"]  # 05 is the primary and the seed source, 06 gets rebuilt
PRIMARY, TARGET = NODES[0], NODES[1]
HOSTS = [f"{n}.{DOMAIN}:27017" for n in NODES]
URI = f"mongodb://{','.join(HOSTS)}/?replicaSet=rs-multi"
DBPATH = "/u01/data/rs-multi"
LOG = "/u01/log/mongod.log"
PG = "aen-mongo-pg-multi"
ROLES = ["data", "journal", "oplog"]

def start(node):
    # first creates the dbPath/journal and dbPath/local symlinks if they aren't there yet
    ssh(node, f"sudo runuser -u mongod -- {BIN}/mongod --replSet rs-multi --bind_ip_all --dbpath {DBPATH} --directoryperdb "
              f"--logpath {LOG} --logappend --oplogSize 990 --wiredTigerCacheSizeGB 2 --fork")

arrays = fleet_arrays(fa)                                 # the FlashArrays in the fleet
vols = {}                                                 # vols[node][role] = (volume, array, serial)
for n in NODES:
    vols[n] = {}
    for role in ROLES:
        serial = volume_serial(n, f"/u01/{role}")         # FlashArray serial of the device behind the mount
        v, a = find_volume(fa, arrays, serial)            # the volume with this serial, and its array
        vols[n][role] = (v, a, serial)
array = {n: vols[n]["data"][1] for n in NODES}            # each node's FlashArray

With that in place, here’s the build. wait_for_primary polls db.hello().isWritablePrimary on aen-mongo-05 until it’s true, because aen-mongo-05 has the higher priority.

for n in NODES:
    start(n)
mongosh(PRIMARY, f"""rs.initiate({{ _id: 'rs-multi', members: [
  {{ _id: 0, host: '{HOSTS[0]}', priority: 2 }}, {{ _id: 1, host: '{HOSTS[1]}' }}, {{ _id: 2, host: '{HOSTS[2]}' }} ] }})""")
wait_for_primary()  # wait until aen-mongo-05 is a writable primary
time.sleep(5)
print(members())    # each member's name and state from rs.status()

mongosh(PRIMARY, """const c = db.getSiblingDB('demo').orders; let batch = [];
for (let i = 0; i < 1000000; i++) {
  batch.push({ _id: i, customer: 'c' + (i % 5000), amount: Math.random() * 100, note: 'x'.repeat(200) });
  if (batch.length === 10000) { c.insertMany(batch); batch = []; }
}""", uri=URI)
time.sleep(10)
print(mongosh(PRIMARY, "rs.printReplicationInfo()", uri=direct(PRIMARY)))

snap = fa.post_protection_group_snapshots(source_names=[PG], context_names=[array[TARGET]],
                                          protection_group_snapshot=ProtectionGroupSnapshotPost(suffix="own06"))
print("Snapshot:", list(snap.items)[0].name)

Output:

aen-mongo-05.fsa.lab:27017  PRIMARY
aen-mongo-06.fsa.lab:27017  SECONDARY
aen-mongo-07.fsa.lab:27017  SECONDARY
actual oplog size
'990 MB'
---
configured oplog size
'990 MB'
---
log length start to end
'29 secs (0.01 hrs)'
...output omitted...
Snapshot: aen-mongo-pg-multi.own06

The snapshot aen-mongo-pg-multi.own06 holds aen-mongo-06’s data, journal, and oplog volumes from right after the load. Look at log length start to end. That’s the oplog window, and it says 29 seconds only because the replica set itself is less than a minute old. More on reading it in Step 2.

Step 2: Restore a Member from Its Own Snapshot

Now, let’s write 50,000 more documents so aen-mongo-06’s snapshot is a little out of date, then take aen-mongo-06 down and put it back from the snapshot with the suffix of own06 (the one we just took): three unmounts, three overwrites, and three mounts. detach stops mongod and unmounts the volumes. attach mounts them by device path (/dev/disk/by-id/wwn-0x624a9370 plus the volume serial) instead of through fstab. transitions reads the member’s JSON log and prints state changes, sync source changes, and any No initial sync required, Too stale, or rollback message. On the array, overwrite_local makes one post_volumes call per volume with the snapshot’s copy as the source:

def overwrite_local(snapshot):
    for role in ROLES:
        volume, arr, _ = vols[TARGET][role]
        r = fa.post_volumes(names=[volume], context_names=[arr], overwrite=True,
                            volume=VolumePost(source=Reference(name=f"{snapshot}.{volume}")))
        if isinstance(r, flasharray.ErrorResponse):  # an overwrite error comes back, it doesn't raise
            sys.exit(f"overwrite {volume} failed: {[e.message for e in r.errors]}")
        print(f"Overwrote {volume} from {snapshot}.{volume}")

mongosh(PRIMARY, "db.getSiblingDB('demo').orders.insertMany(Array.from({length: 50000}, "
                 "(_, i) => ({ _id: 20000000 + i, note: 'later' })))", uri=URI)
since = ssh(TARGET, "date -u +%Y-%m-%dT%H:%M:%S")    # aen-mongo-06's clock, so I read only the log lines from here on

detach(TARGET)                                       # stop mongod, unmount data, journal, and oplog
overwrite_local(f"{PG}.own06")
attach(TARGET)                                       # mount data, journal, and oplog by device path
start(TARGET)
time.sleep(15)
print(members())                                     # each member's name and state from rs.status()
print("06:", count(TARGET), " 05:", count(PRIMARY))  # documents in a demo collection, read on that node
print(transitions(TARGET, since))                    # state changes from the member's log since 'since'

Output:

Overwrote aen-mongo-multi-06-data from aen-mongo-pg-multi.own06.aen-mongo-multi-06-data
Overwrote aen-mongo-multi-06-journal from aen-mongo-pg-multi.own06.aen-mongo-multi-06-journal
Overwrote aen-mongo-multi-06-oplog from aen-mongo-pg-multi.own06.aen-mongo-multi-06-oplog
aen-mongo-05.fsa.lab:27017  PRIMARY
aen-mongo-06.fsa.lab:27017  SECONDARY
aen-mongo-07.fsa.lab:27017  SECONDARY
06: 1050000  05: 1050000
2026-11-16T23:22:08   STARTUP -> STARTUP2
2026-11-16T23:22:08   No initial sync required. Attempting to begin steady replica
2026-11-16T23:22:08   STARTUP2 -> RECOVERING
2026-11-16T23:22:08   RECOVERING -> SECONDARY
2026-11-16T23:22:09   syncing from aen-mongo-05

aen-mongo-06 came back as a secondary, and it holds the same 1,050,000 documents as the primary: the million from before the snapshot plus the 50,000 I wrote after it. Its log shows it went from STARTUP to SECONDARY inside one second, logged No initial sync required on the way, and a second later it was syncing from aen-mongo-05 to pick up those 50,000 documents. The overwrite put back the bulk of the data as a metadata operation on the array, and the oplog filled in the rest.

How a Restored Member Gets Back in Sync

From aen-mongo-06’s point of view, a restore from a snapshot looks a lot like someone tripping over its power cord…it just picks up where replication left off. The only difference is which moment it wakes up in:

  1. Crash recovery: WiredTiger opens the data files at their last stable checkpoint and replays the journal, which in a replica set covers the local database and its oplog. That’s why the data, journal, and oplog share one protection group: all three have to describe the same moment.
  2. Replication recovery: mongod replays its own oplog into your collections, from that checkpoint up to the moment of the snapshot.
  3. Rejoining: It finds its replica set configuration, logs No initial sync required, and moves from STARTUP2 to RECOVERING to SECONDARY.
  4. Catching up: It picks a sync source whose oplog still has its last operation, here syncing from aen-mongo-05, and tails it from there. That’s ordinary replication, and it only has to cover the time since the snapshot.

The key for this snapshot scenario is stage 4. Some member’s oplog, not necessarily the primary’s, has to reach all the way back to the moment of the snapshot. If every member has already rolled past it, there’s nothing left to catch up from.

Keep Your Snapshots Inside the Oplog Window

So you need to know how much oplog you’ll need:

  • Read the window: rs.printReplicationInfo() on a member prints log length start to end, the time between its oldest and newest oplog entries. For the exact edge, read the oldest entry itself with db.getSiblingDB('local').oplog.rs.find().sort({ $natural: 1 }).limit(1).next().wall.
  • Check every possible sync source: Each member truncates its own oplog, so the window is per member, and a restored member catches up from any member that still reaches back far enough. My first try at the next demo only waited on the primary’s oplog, and aen-mongo-06 just denylisted aen-mongo-05 and caught up from aen-mongo-07.
  • Watch time, not size: The configured size isn’t a hard ceiling. My 990 MB oplog sat at 1,557 MB during this testing.
  • Put the snapshots on a schedule: Give the protection group a snapshot schedule, say every hour, and set the oplog’s minimum retention longer than the interval plus a buffer. That’s --oplogMinRetentionHours at startup, or replSetResizeOplog with minRetentionHours on a running member. Then a member’s most recent snapshot is always inside the window, and this restore always works. The oplog grows during write bursts to honor the retention, so size the oplog volume with that in mind.

What It Looks Like When You’re Outside the Window

Now let’s break it on purpose. With aen-mongo-06 down, I wrote to the primary until the oldest entry on both possible sync sources, aen-mongo-05 and aen-mongo-07, was newer than own06, then restored aen-mongo-06 from it again:

since = ssh(TARGET, "date -u +%Y-%m-%dT%H:%M:%S")
# detach(TARGET), then keep writing until the oldest oplog entry on 05 AND 07 is newer than own06
overwrite_local(f"{PG}.own06")
attach(TARGET)                     # mount data, journal, and oplog by device path
start(TARGET)
time.sleep(30)
print(members())                   # each member's name and state from rs.status()
print(transitions(TARGET, since))  # state changes from the member's log since 'since'

Output:

aen-mongo-05.fsa.lab:27017  PRIMARY
aen-mongo-06.fsa.lab:27017  RECOVERING
aen-mongo-07.fsa.lab:27017  SECONDARY
2026-11-16T23:24:00   STARTUP -> STARTUP2
2026-11-16T23:24:00   No initial sync required. Attempting to begin steady replica
2026-11-16T23:24:00   STARTUP2 -> RECOVERING
2026-11-16T23:24:00   RECOVERING -> SECONDARY
2026-11-16T23:24:01   too stale to sync from aen-mongo-05
2026-11-16T23:24:01   too stale to sync from aen-mongo-07
2026-11-16T23:24:01   Too stale to catch up. Entering maintenance mode. See http:/
2026-11-16T23:24:01   SECONDARY -> RECOVERING

aen-mongo-06 logs No initial sync required and even reaches SECONDARY at 23:24:00. One second later it’s too stale for both sync sources, logs Too stale to catch up, and drops back to RECOVERING, where it stays. Two lessons here. No initial sync required only means mongod found data and an oplog on disk, and the proof that a member is healthy is that it stays in SECONDARY. And from here, MongoDB’s only way forward is an initial sync. That’s the job the next step replaces.

Step 3: Reseed from a Snapshot of the Primary, with No fsyncLock

The fix for a member that’s too stale, or brand new, is a fresh copy of a healthy member. Here that’s the primary, whose protection group replicates to aen-mongo-06’s array. The usual advice is to fsyncLock the primary first, and MongoDB’s filesystem snapshot documentation asks for it whenever the journal is on a different volume from the data files. The lock blocks every write to the replica set while it’s held.

You don’t need it here. The lock is there because separate snapshots of separate volumes don’t describe the same moment. A protection group snapshot takes the primary’s data, journal, and oplog at the same instant, so it’s the same crash-consistent image as in the last post, and the four stages above are how a member recovers from one. So I snapshot the primary under load and don’t lock anything:

since = ssh(TARGET, "date -u +%Y-%m-%dT%H:%M:%S")
start_load()                     # four writers with their own sequence numbers, plus an update stream, on the primary
time.sleep(15)
rate = ops_per_second()          # the primary's inserts plus updates, sampled over five seconds
t0 = time.time()
snap = fa.post_protection_group_snapshots(source_names=[PG], context_names=[array[PRIMARY]], replicate_now=True,
                                          protection_group_snapshot=ProtectionGroupSnapshotPost(suffix="primary05"))
name = list(snap.items)[0].name  # aen-mongo-pg-multi.primary05
snap_secs = time.time() - t0
rate_after = ops_per_second()    # primary's inserts plus updates per second, over 5 s

replicate_now=True forces the replication at the moment of the snapshot. The transfer to flasharray02 starts as soon as the snapshot exists, without waiting for the replication schedule, and the snapshot’s transfer record gets a completed time when it’s done. On the target array, a replicated snapshot carries the source array’s name in front: <source array>:<protection group>.<suffix>. Once the transfer completes, I overwrite aen-mongo-06’s volumes from aen-mongo-05’s, matched by role, and start aen-mongo-06 while the load keeps running.

while not getattr(list(fa.get_protection_group_snapshots_transfer(filter=f"name='{name}'",
                                                                  context_names=[array[PRIMARY]]).items)[0],
                  "completed", None):
    time.sleep(5)
replicated = time.time() - t0

detach(TARGET)  # stop mongod, unmount data, journal, and oplog
for role in ROLES:
    vol, arr, _ = vols[TARGET][role]
    src = vols[PRIMARY][role][0]
    r = fa.post_volumes(names=[vol], context_names=[arr], overwrite=True,
                        volume=VolumePost(source=Reference(name=f"{array[PRIMARY]}:{name}.{src}")))
    if isinstance(r, flasharray.ErrorResponse):
        sys.exit(f"overwrite {vol} failed: {[e.message for e in r.errors]}")
    print(f"Overwrote {vol} from {array[PRIMARY]}:{name}.{src}")
attach(TARGET)  # mount data, journal, and oplog by device path
start(TARGET)

Then the checks. After aen-mongo-06 reaches SECONDARY, the load runs for 15 more seconds so aen-mongo-06 has to keep up, not just show up. Then the load stops, every member has to reach the same last operation, and 20 seconds later aen-mongo-06 has to still be a secondary. Then the dbHash of the demo database has to match on all three members (the output shows the first 12 characters), and no writer can have a gap on aen-mongo-06. Each writer numbers its documents from 0, so a writer with n documents should top out at sequence n - 1.

# state(node): the member's state from rs.status()    dbhash(node): the md5 from dbHash on demo
# stop_load(): touch the writers' stop file    gaps(node): per-writer count against the highest seq
# wait_until_caught_up(): poll rs.status() until every member reports the same optime
while state(TARGET) != "SECONDARY":            # that member's state from rs.status()
    time.sleep(2)
back = time.time() - t0
time.sleep(15)                                 # the load keeps running: aen-mongo-06 has to keep up, not just arrive
stop_load()                                    # tell the writers and updater to stop
caught_up = wait_until_caught_up()             # wait until every member reports the same optime
time.sleep(20)
still = state(TARGET)                          # that member's state from rs.status()

hashes = {n: dbhash(n) for n in NODES}         # md5 of dbHash on demo, read on that member
g = gaps(TARGET)                               # check each writer's sequence for gaps
counts = {n: count(n, "load") for n in NODES}  # documents in a demo collection, read on that node
log = transitions(TARGET, since)               # state changes from the member's log since 'since'
ok = (caught_up and still == "SECONDARY" and len(set(hashes.values())) == 1 and "GAPS" not in g
      and len(set(counts.values())) == 1 and "Too stale" not in log and "rollback" not in log.lower())

# prints the snapshot timing, the load rate, the members, the counts, the hashes, the gap check, and aen-mongo-06's log, shown below
print("PASS" if ok else "FAIL")

Output:

Overwrote aen-mongo-multi-06-data from flasharray01:aen-mongo-pg-multi.primary05.aen-mongo-multi-05-data
Overwrote aen-mongo-multi-06-journal from flasharray01:aen-mongo-pg-multi.primary05.aen-mongo-multi-05-journal
Overwrote aen-mongo-multi-06-oplog from flasharray01:aen-mongo-pg-multi.primary05.aen-mongo-multi-05-oplog
Snapshot: aen-mongo-pg-multi.primary05, no fsyncLock, 0.3 seconds for the snapshot call
Load on the primary: 3296 ops/s before the snapshot, 3373 ops/s right after
Replicated to flasharray02 in 24 seconds; member 06 SECONDARY 52 seconds after the snapshot
aen-mongo-05.fsa.lab:27017  PRIMARY
aen-mongo-06.fsa.lab:27017  SECONDARY
aen-mongo-07.fsa.lab:27017  SECONDARY
load documents: 05: 198100  06: 198100  07: 198100
demo dbHash:    05: 74600695b9e0  06: 74600695b9e0  07: 74600695b9e0
writer 0: 50800 documents, highest seq 50799, no gaps
writer 1: 49700 documents, highest seq 49699, no gaps
writer 2: 49200 documents, highest seq 49199, no gaps
writer 3: 48400 documents, highest seq 48399, no gaps
2026-11-16T23:29:38   STARTUP -> STARTUP2
2026-11-16T23:29:38   No initial sync required. Attempting to begin steady replica
2026-11-16T23:29:38   STARTUP2 -> RECOVERING
2026-11-16T23:29:38   RECOVERING -> SECONDARY
2026-11-16T23:29:39   syncing from aen-mongo-05
member 06 still SECONDARY 20 seconds after the load stopped
PASS

aen-mongo-06 was a secondary again 52 seconds after the snapshot, with no lock and no initial sync. The snapshot call took 0.3 seconds, and the primary was doing about 3,300 operations a second before it and about 3,400 right after, so nothing paused. Look at the Overwrote lines: aen-mongo-06’s volumes now come from aen-mongo-05’s, and the flasharray01: prefix says they’re the replicated copy on flasharray02. All three members have the same dbHash and the same 198,100 load documents, and every writer’s sequence is unbroken. The 24 seconds of replication includes a five second throughput sample and five second polling, so the raw transfer was shorter.

Doing It Again, and Again

One clean run is a demo. If I’m going to tell you that you don’t need fsyncLock on the primary, it has to work every time. So I put the reseed in a loop: reseed(suffix, verbose=False) is everything in Step 3 above in one function, returning PASS or FAIL and that run’s numbers, and each run destroys its snapshots afterward.

runs = 10
for i in range(1, runs + 1):
    result, d = reseed(f"seed05-{i:02d}", verbose=False)  # Step 3 as one function: PASS or FAIL plus the numbers
    print(f"run {i:2d}: {result}  {d['rate']} ops/s before, {d['rate_after']} after, snapshot call {d['snap']}s, "
          f"replicated {d['repl']}s, SECONDARY after {d['back']}s, {d['docs']} load docs, "
          f"hashes match: {d['hash']}, no gaps: {d['gaps']}, still {d['still']}", flush=True)
    # destroyed, not eradicated
    for a, n in ((array[PRIMARY], f"{PG}.seed05-{i:02d}"), (array[TARGET], f"{array[PRIMARY]}:{PG}.seed05-{i:02d}")):
        fa.patch_protection_group_snapshots(names=[n], context_names=[a],
                                            protection_group_snapshot=ProtectionGroupSnapshotPatch(destroyed=True))

Each run prints one long line. I’ve wrapped them here so they fit the page:

Output:

run  1: PASS  3306 ops/s before, 3446 after,
        snapshot call 0.3s, replicated 19s, SECONDARY after 49s,
        190900 load docs, hashes match: True, no gaps: True, still SECONDARY
run  2: PASS  3256 ops/s before, 3381 after,
        snapshot call 0.2s, replicated 19s, SECONDARY after 50s,
        191900 load docs, hashes match: True, no gaps: True, still SECONDARY
...output omitted...
run 10: PASS  3308 ops/s before, 3411 after,
        snapshot call 0.3s, replicated 13s, SECONDARY after 43s,
        179700 load docs, hashes match: True, no gaps: True, still SECONDARY

That’s ten reseeds in a row, and 10 of 10 passed, on top of the run above. Across all ten:

Measure Range across the 10 runs
Load on the primary before the snapshot 3,178 to 3,334 ops/s
Load on the primary right after the snapshot 3,303 to 3,600 ops/s
Snapshot call 0.2 to 0.3 seconds
Replication to flasharray02 13 to 19 seconds
Snapshot to aen-mongo-06 in SECONDARY 40 to 50 seconds
Load documents written per run 170,000 to 191,900
Matching dbHash, no gaps, still SECONDARY 10 of 10

In every run, the primary did more work in the five seconds after the snapshot call than in the five seconds before it. With no fsyncLock, nothing holds writes.

Rebuilding Every Secondary from One Snapshot

If one snapshot of the primary can rebuild aen-mongo-06, it can rebuild every secondary. The only new piece is replication. Each secondary’s array needs its own copy of the primary’s snapshot, so I add flasharray03, aen-mongo-07’s array, as a second target of aen-mongo-05’s protection group. Then one snapshot with replicate_now=True sends a copy to both arrays at once.

fa.post_protection_groups_targets(group_names=[PG], member_names=["flasharray03"], context_names=[array[PRIMARY]])

start_load()                                             # four writers and an updater on the primary
snap = fa.post_protection_group_snapshots(source_names=[PG], context_names=[array[PRIMARY]], replicate_now=True,
                                          protection_group_snapshot=ProtectionGroupSnapshotPost(suffix="all05"))
remote = f"{array[PRIMARY]}:{list(snap.items)[0].name}"  # flasharray01:aen-mongo-pg-multi.all05
# wait until the replicated copy shows up on flasharray02 and flasharray03, then rebuild one secondary at a time
for n in ["aen-mongo-06", "aen-mongo-07"]:
    detach(n)                                            # stop mongod, unmount data, journal, and oplog
    for role in ROLES:
        v, a, _ = vols[n][role]
        r = fa.post_volumes(names=[v], context_names=[a], overwrite=True,
                            volume=VolumePost(source=Reference(name=f"{remote}.{vols[PRIMARY][role][0]}")))
        if isinstance(r, flasharray.ErrorResponse):
            sys.exit(f"overwrite {v} failed: {[e.message for e in r.errors]}")
    attach(n)                                            # mount data, journal, and oplog by device path
    start(n)
    while state(n) != "SECONDARY":                       # that member's state from rs.status()
        time.sleep(2)

One at a time is the important part. rs-multi has three voting members, so a majority is two. If I stopped both secondaries together, aen-mongo-05 would lose its majority, step down, and every write would stop. One after the other keeps two members up the whole time. After the second rebuild, the same checks as before run on all three members.

Output:

Snapshot: aen-mongo-pg-multi.all05, no fsyncLock, 0.3 seconds for the snapshot call
Load on the primary: 2976 ops/s before the snapshot, 3061 ops/s right after
  flasharray01:aen-mongo-pg-multi.all05 on flasharray02 after 8 s
  flasharray01:aen-mongo-pg-multi.all05 on flasharray03 after 8 s
aen-mongo-06 rebuilt from flasharray01:aen-mongo-pg-multi.all05 on flasharray02: SECONDARY 30 s after it stopped, primary still PRIMARY
aen-mongo-07 rebuilt from flasharray01:aen-mongo-pg-multi.all05 on flasharray03: SECONDARY 30 s after it stopped, primary still PRIMARY
aen-mongo-05.fsa.lab:27017  PRIMARY
aen-mongo-06.fsa.lab:27017  SECONDARY
aen-mongo-07.fsa.lab:27017  SECONDARY
load documents: 05: 237700  06: 237700  07: 237700
demo dbHash:    05: e78d3a76c6d1  06: e78d3a76c6d1  07: e78d3a76c6d1
writer 0: 60200 documents, highest seq 60199, no gaps
...output omitted...
--- aen-mongo-07
2026-11-17T12:57:42   STARTUP -> STARTUP2
2026-11-17T12:57:42   No initial sync required. Attempting to begin steady replica
2026-11-17T12:57:42   STARTUP2 -> RECOVERING
2026-11-17T12:57:42   RECOVERING -> SECONDARY
2026-11-17T12:57:43   syncing from aen-mongo-05
PASS

One snapshot of the primary rebuilt every secondary while the load kept running. The replicated copy showed up on both arrays in the same poll, 8 seconds after the snapshot (five of those seconds were the throughput sample), because replicate_now started both transfers at once. Each secondary was back 30 seconds after it stopped, and every rebuild line ends with primary still PRIMARY. All three members finished with the same 237,700 load documents and the same dbHash, e78d3a76c6d1.

Two Details from the Copy

A copy of another member’s volumes brings two things along:

  • File system UUIDs: The rebuilt members’ file systems are now copies of aen-mongo-05’s, so they carry aen-mongo-05’s UUIDs and no longer match their own fstab. That’s why attach mounts them by device path. If you reseed for real, update fstab afterward so the mounts survive a reboot.
  • Writes the primary hadn’t replicated yet: A snapshot of the primary can include writes that hadn’t reached a majority of the members at that instant. As long as aen-mongo-05 stays primary, those writes get majority-committed and nothing changes. If aen-mongo-05 failed before they replicated and another member became primary without them, a rebuilt member would roll them back, the same way any other member holding those writes would.

Why This Matters

  1. A replicated snapshot replaces the initial sync: An initial sync grows with your data. A replicated snapshot sends only the blocks that changed since the last one, and the snapshot and the overwrite take the same time whether the database is 10 GB or 10 TB.
  2. You can seed from the primary without freezing it: With data, journal, and oplog in one protection group, the primary’s snapshot is crash-consistent on its own. Across 11 reseeds under load, nothing held writes.
  3. A member’s own snapshot comes first: A snapshot schedule plus a minimum oplog retention longer than the interval keeps the most recent snapshot inside the window.
  4. “No initial sync required” doesn’t mean it worked: A member outside the window logs it too, then fails with Too stale to catch up. The proof is a member that stays in SECONDARY.
  5. The log is on your side: Keeping it out of the protection group let me show you every member’s state changes after every restore, especially helpful in the reseeding scenarios.

Wrapping Up

The bottom line is that rebuilding a member doesn’t have to mean an initial sync. Inside the oplog window, a member’s own snapshot puts it back with a short oplog replay, and a snapshot schedule keeps you inside that window. Outside it, or for a new member, a replicated snapshot of the primary does the job, taken under load with no fsyncLock, and the same snapshot rebuilds every secondary in the set.

Two things to keep in your head when you do this yourself. First, know your oplog window on every member before the day you need it, and watch a restored member after it rejoins, because it can reach SECONDARY and still fall over a second later. Second, a cross-member copy brings its file system UUIDs with it, so plan for fstab. MongoDB’s guide to resyncing a replica set member is worth reading alongside this. Next, I’ll show you when you do need the lock: restoring every member from its own array’s local snapshot, and what goes wrong without it. Get out in your lab and try it, and let me know how it works in your environment.

MongoDB on Everpure Series

  1. Why Run MongoDB on Everpure FlashArray?
  2. Snapshot Recovery of a MongoDB Replica Set on a Single FlashArray
  3. Replacing the MongoDB Replica Set Initial Sync with FlashArray Snapshots (This Post)
  4. When You Need fsyncLock: Restoring a MongoDB Replica Set from Local Snapshots on Every FlashArray
  5. Expanding a MongoDB Replica Set with Everpure Fusion
  6. Where FlashBlade Fits When MongoDB Lives on FlashArray
  7. A Tag-Based Snapshot Catalog for a MongoDB Replica Set Across FlashArrays with Everpure Fusion

Related: App-Consistent MongoDB Snapshots Across Multiple Pure Storage FlashArrays, my earlier post on snapshotting a sharded MongoDB cluster across two arrays.