When You Need fsyncLock: Restoring a MongoDB Replica Set from Local Snapshots on Every FlashArray
Page content
In my reseed post, I spent a lot of time convincing you that you don’t need fsyncLock. One crash-consistent snapshot of the primary, replicated to every array, rebuilt every secondary in the set, under load, with no pause in writes. That’s still what I reach for when I’m rebuilding members.
Then I asked myself the obvious follow-up: what if I want to put every member back to the same known moment, each one from its own array’s local snapshot? That’s the one case in this series where I do reach for the lock. It’s also the Python version of my 2025 post on app-consistent snapshots across multiple FlashArrays, this time for a replica set instead of a sharded cluster.
In this post, I’m going to show you how I freeze the primary for a few seconds, snapshot all three arrays, and restore every member back to the exact document count I recorded under the lock. Then I’ll show you what happened when I ran the same test without the lock, because the result looks fine right up until you read the logs. Let’s go.
The Problem We’re Solving
A protection group snapshot is atomic on one array. My replica set lives on three. Restoring each member from its own array’s snapshot is attractive. Nothing moves between arrays at restore time, every overwrite is a metadata operation on that member’s own array, and you don’t depend on a replicated copy being there and recent enough.
The catch is time. Those are three snapshots on three arrays, taken one after another while the replica set keeps writing. Each one is crash-consistent on its own, but together they aren’t one moment. When the bad day is someone dropping a database, here’s what I want from the restore:
- Every member at the same point: Replication has nothing to sort out after the restore.
- A number I knew in advance: I record a document count at the moment of the snapshot, so the restore can prove itself.
- The shortest possible pause: Whatever I do to writes, I want it measured in seconds.
Freezing the primary with fsyncLock for the few seconds the snapshots take gives me all three. I tested what happens without it, too, and the answer is worse than “the members land at slightly different points.”
When a Lock Is Required
This post covers one of four cases where you need a lock:
- Every member restored from its own array’s snapshot: The snapshots are separate calls to separate arrays, so they aren’t one moment, and without a lock the members come back out of step. That’s this post.
- A member’s volumes can’t be captured atomically: If one
mongod’s data, journal, and oplog sit in different protection groups or on different arrays, even that member’s snapshot isn’t crash-consistent. MongoDB’s documentation is direct about this one: with the journal on a separate volume, you lock the database during the backup. This series keeps each member’s three volumes in one protection group, and that’s what removes the requirement everywhere else. - A sharded cluster: You stop the balancer and lock through
mongos, so chunk migrations and the config servers line up with the shards. That’s the 2025 post. - Separate replica sets that must agree: If an application writes to two replica sets together and you have to restore both to the same moment, you’ve got the sharding problem without a
mongosto coordinate it.
Here’s how the local-snapshot approach stacks up against the no-lock approach from the reseed post:
| One snapshot of the primary, replicated everywhere | Each member’s own local snapshot | |
|---|---|---|
| Pause in writes | None | A few seconds, while the snapshots are taken |
| Data moved between arrays | One replicated copy per array, ahead of time | None |
| Restore depends on | The replicated copy being there and recent enough | Only each array’s own snapshot |
| What you know about the restore point | A window, like the single-array post | The exact opTime and document count |
My Lab Setup
I’m using the same three-member rs-multi replica set from the reseed post, one member per array, with aen-mongo-05 as the primary. Every member has the same split layout, and each array’s aen-mongo-pg-multi holds that member’s data, journal, and oplog volumes. The log volume stays out of the group. If fsyncLock or opTimes are new to you, the terms list in the intro post covers them.
aen-mongo-05 aen-mongo-06 aen-mongo-07
PRIMARY SECONDARY SECONDARY
flasharray01 flasharray02 flasharray03
aen-mongo-pg-multi on each array: that member's data, journal, and oplog
The code reuses the helpers from the reseed post. ssh and mongosh run commands on a host, and direct(node) builds a direct connection string to one member. detach(node) stops mongod and unmounts the data, journal, and oplog, attach(node) mounts them again by device path, and start(node) starts mongod with the same flags and symlinks. vols[node][role] holds each member’s volume names, found by serial.
What You’ll Need
Here’s what I’m running, and what you’ll need to follow along:
- FlashArrays: Three here, one per member, running Purity//FA 6.10 and 6.12 in my lab.
- An Everpure Fusion fleet: All three arrays are joined to one fleet, so a single login to the gateway array reaches every one of them. I route each call to the right array with
context_names. - A protection group with the same name on each array: Each one holds that member’s data, journal, and oplog volumes, and leaves its log volume out.
- A replica set with one member per array: MongoDB Enterprise 8.0.21 and
mongosh2.8.1 on RHEL 9.6 in my lab. - A control machine: Python 3 with
py-pure-client1.99.0, SSH key access to each host as a user with passwordless sudo, and array credentials inFA_ENDPOINT,FA_USERNAME, andFA_PASSWORD.
Step 1: Freeze the Primary and Snapshot Every Array
Freezing an idle database doesn’t prove much, so first I start a writer on the primary.
suffix = "lock-" + datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
js = f"""const c = db.getSiblingDB('demo').orders; let s = 0;
while (true) {{ c.insertOne({{ _id: '{suffix}-' + s, w: '{suffix}', seq: s }}); s++;
if (s % 50 === 0 && require('fs').existsSync('/tmp/stop-appcon')) break; }}"""
b64 = base64.b64encode(js.encode()).decode()
ssh(PRIMARY, f"nohup bash -c 'mongosh --quiet \"{URI}\" --eval \"$(echo {b64} | base64 -d)\"' </dev/null >/tmp/appcon.log 2>&1 &")
time.sleep(10)
Now the freeze. I lock the primary only. All writes in a replica set go through the primary, so that one lock blocks every write to the set. fsyncLock stops writes, not reads, so the secondaries keep pulling and applying the primary’s oplog until every member reports the same opTime. They don’t need a lock of their own, because each member’s snapshot is crash-consistent by itself.
Every mongosh call here is a fresh SSH session that costs about a second, so the lock, the wait for every member to catch up, and the counts all happen in one call. The three snapshots go in a try and the unlock goes in a finally, so a failed snapshot call can’t leave the replica set unable to write.
MEMBERS = {"aen-mongo-05": "flasharray01", "aen-mongo-06": "flasharray02", "aen-mongo-07": "flasharray03"}
t0 = time.time()
try:
frozen = json.loads(mongosh(PRIMARY, f"""const lock = db.fsyncLock();
let t; do {{ t = rs.status().members.map(m => m.optime.ts.t + '.' + m.optime.ts.i); if (new Set(t).size > 1) sleep(100); }} while (new Set(t).size > 1);
const d = db.getSiblingDB('demo');
print(JSON.stringify({{ lockCount: Number(lock.lockCount), opTime: t[0],
orders: d.orders.countDocuments(), writer: d.orders.countDocuments({{ w: '{suffix}' }}) }}))""", uri=direct(PRIMARY)))
for n, a in MEMBERS.items():
fa.post_protection_group_snapshots(source_names=[PG], context_names=[a],
protection_group_snapshot=ProtectionGroupSnapshotPost(suffix=suffix))
finally:
unlock = mongosh(PRIMARY, "print(Number(db.fsyncUnlock().lockCount))", uri=direct(PRIMARY))
print(f"writes blocked for {time.time() - t0:.1f} s") # plus the lockCount, opTime, and counts shown below
Output:
fsyncLock on aen-mongo-05: lockCount 1
all 3 members at opTime 1794920950.131: orders 1121335, writer 8261
Snapshot on flasharray01: aen-mongo-pg-multi.lock-20261117-130857
Snapshot on flasharray02: aen-mongo-pg-multi.lock-20261117-130857
Snapshot on flasharray03: aen-mongo-pg-multi.lock-20261117-130857
fsyncUnlock: lockCount 0
lock and catch-up 2.3 s, 3 snapshot calls 0.9 s, writes blocked for 4.3 s in total
writer after the unlock: 17050 documents
Let’s walk through it top to bottom. lockCount 1 means the primary is locked. All three members agreed on opTime 1794920950.131, and while the lock was held, I read 1,121,335 orders, 8,261 of them from this run’s writer. Those counts are the bar the restore has to hit later. Next come three snapshots on three arrays, all with one name, aen-mongo-pg-multi.lock-20261117-130857. Then lockCount 0 after the unlock, so writes are flowing again. The last line is the proof the writer was alive through the whole thing: it went from 8,261 to 17,050 documents, more than doubling in the roughly 10 seconds after the unlock.
The three snapshots took 0.9 seconds, and writes were blocked for at most 4.3 seconds, measured from sending the lock to the unlock returning. Most of that is the two SSH round trips for the lock and the unlock, not the arrays or MongoDB. A tool that holds one MongoDB connection open would hold the lock for less.
Step 2: The Disaster and the Restore
Let’s do something bad and drop the demo database on purpose. The drop replicates to every member, the way it would in real life, so there’s no healthy secondary to fall back on. Then I stop the secondaries first and the primary last, so there’s no election while we do this work. Let’s overwrite each member’s data, journal, and oplog from the snapshot on its own array, mount everything again, and start the primary first:
mongosh(PRIMARY, "db.getSiblingDB('demo').dropDatabase()", uri=URI)
t0 = time.time()
for n in ["aen-mongo-06", "aen-mongo-07", "aen-mongo-05"]:
detach(n) # stop mongod, unmount data, journal, and oplog
for n, a in MEMBERS.items():
for role in ROLES:
v = vols[n][role][0]
r = fa.post_volumes(names=[v], context_names=[a], overwrite=True,
volume=VolumePost(source=Reference(name=f"{PG}.{suffix}.{v}")))
if isinstance(r, flasharray.ErrorResponse): # an overwrite error comes back, it doesn't raise
sys.exit(f"overwrite {v} failed: {[e.message for e in r.errors]}")
for n in ["aen-mongo-05", "aen-mongo-06", "aen-mongo-07"]:
attach(n) # mount data, journal, and oplog by device path
start(n) # mongod with the same flags and symlinks
while "true" not in mongosh(PRIMARY, "print(db.hello().isWritablePrimary)", uri=direct(PRIMARY), allow_fail=True):
time.sleep(2)
print(f"Restore finished in {time.time() - t0:.0f} seconds, from the first shutdown to {PRIMARY} PRIMARY")
Step 3: Verify Against the Frozen Counts
Let’s check the document counts. For each member, I read the orders count and this run’s writer documents straight from that member, check that the writer’s sequence has no gaps, and compare the dbHash of the demo database. Then I check the replica set’s state again 20 seconds later, and finally I check each member’s mongod log for anything that happened since the restore started.
def counts(node, tag):
return json.loads(mongosh(node, f"""db.getMongo().setReadPref('secondaryPreferred');
const d = db.getSiblingDB('demo');
const w = d.orders.aggregate([{{ $match: {{ w: '{tag}' }} }}, {{ $group: {{ _id: null, n: {{ $sum: 1 }}, max: {{ $max: '$seq' }} }} }}]).toArray()[0] || {{ n: 0, max: -1 }};
print(JSON.stringify({{ orders: d.orders.countDocuments(), writer: w.n, maxSeq: w.max }}))""", uri=direct(node)))
c = {n: counts(n, suffix) for n in MEMBERS}
h = {n: mongosh(n, "print(db.getSiblingDB('demo').runCommand({ dbHash: 1 }).md5)", uri=direct(n)) for n in MEMBERS}
ok = (all(c[n]["orders"] == frozen["orders"] and c[n]["writer"] == frozen["writer"] for n in MEMBERS)
and all(c[n]["writer"] == c[n]["maxSeq"] + 1 for n in MEMBERS) and len(set(h.values())) == 1)
# prints each member's counts and hash, checks the state again 20 seconds later, then PASS or FAIL
# (the log check at the end of the output is a separate grep of each member's mongod log)
Output:
Restore finished in 73 seconds, from the first shutdown to aen-mongo-05 PRIMARY
aen-mongo-05.fsa.lab:27017 PRIMARY
aen-mongo-06.fsa.lab:27017 SECONDARY
aen-mongo-07.fsa.lab:27017 SECONDARY
frozen at the snapshot: orders 1121335, writer 8261
aen-mongo-05: orders 1121335, writer 8261 (highest seq 8260, no gaps), dbHash cbd49cf698c1
aen-mongo-06: orders 1121335, writer 8261 (highest seq 8260, no gaps), dbHash cbd49cf698c1
aen-mongo-07: orders 1121335, writer 8261 (highest seq 8260, no gaps), dbHash cbd49cf698c1
20 seconds later:
aen-mongo-05.fsa.lab:27017 PRIMARY
aen-mongo-06.fsa.lab:27017 SECONDARY
aen-mongo-07.fsa.lab:27017 SECONDARY
PASS
Log check since the restore started (2026-11-17T13:09:15 UTC), per member:
aen-mongo-05: unclean-shutdown recovery 1, 'No initial sync required' 1, rollback or too stale 0
aen-mongo-06: unclean-shutdown recovery 1, 'No initial sync required' 1, rollback or too stale 0
aen-mongo-07: unclean-shutdown recovery 1, 'No initial sync required' 1, rollback or too stale 0
Every member came back with exactly 1,121,335 orders and exactly 8,261 writer documents, the numbers I recorded under the lock. The highest seq 8260 with no gaps means the writer’s sequence runs from 0 to 8,260 with nothing missing on any member. The dbHash (the output shows the first 12 characters) is cbd49cf698c1 on all three, so the members don’t just have the same counts, they have the same data. Then look at the log check. Each member did one crash recovery, which is what a crash-consistent snapshot asks for, logged No initial sync required, and had zero rollbacks. Three members on three arrays, each restored from its own local snapshot, all back at the point I recorded under the lock, in 73 seconds.
What Happens Without the Lock
I wanted to know what happens when I run the same test with no fsyncLock: the same live writer, the same three local snapshots taken one after another, and the same drop and restore. So I did. This time I also read each snapshot’s creation time back from its array, and I recorded the writer’s count right before the first snapshot call and right after the last one. There are three distinct snapshots, one on each array, and inside each snapshot is a slightly different version of the database since the snapshots aren’t atomic across three arrays and we didn’t use fsyncLock to freeze writes into the database.
Output:
Snapshot on flasharray01: aen-mongo-pg-multi.nolock-20261117-134717, created 0 ms after the first
Snapshot on flasharray02: aen-mongo-pg-multi.nolock-20261117-134717, created 197 ms after the first
Snapshot on flasharray03: aen-mongo-pg-multi.nolock-20261117-134717, created 452 ms after the first
no fsyncLock: 3 snapshot calls took 0.9 s, writer at 8688 before the first call and 10611 after the last
writer after the snapshots: 18950 documents
Look at the created times. The three snapshots are spread across 452 milliseconds, and the writer’s count went from 8,688 before the first call to 10,611 after the last, so each member’s copy stops at a different point. After the restore, the set does come back together:
Output:
every member at opTime 1794923350.2
writer window around the snapshot calls: 8688 to 10611
aen-mongo-05: orders 1130502, writer 9167 (highest seq 9166, no gaps), dbHash ac8d40fcd00a
aen-mongo-06: orders 1130502, writer 9167 (highest seq 9166, no gaps), dbHash ac8d40fcd00a
aen-mongo-07: orders 1130502, writer 9167 (highest seq 9166, no gaps), dbHash ac8d40fcd00a
20 seconds later:
aen-mongo-05.fsa.lab:27017 PRIMARY
aen-mongo-06.fsa.lab:27017 SECONDARY
aen-mongo-07.fsa.lab:27017 SECONDARY
PASS
Log check since the restore started (2026-11-17T13:47:55 UTC), per member:
aen-mongo-05: unclean-shutdown recovery 1, 'No initial sync required' 1, rollback or too stale 0
aen-mongo-06: unclean-shutdown recovery 1, 'No initial sync required' 1, rollback or too stale 0
aen-mongo-07: unclean-shutdown recovery 1, 'No initial sync required' 1, rollback or too stale 1
Every member agrees on the opTime, the counts, and the dbHash, the writer’s sequence has no gaps, and the writer’s 9,167 is inside the 8,688 to 10,611 window. The convergence check even printed PASS. It looks fine. Now look at the last line: rollback or too stale 1 on aen-mongo-07, the member whose snapshot was taken last. So I pulled aen-mongo-07’s state changes from its log:
Output:
2026-11-17T13:49:10 STARTUP -> STARTUP2
2026-11-17T13:49:10 STARTUP2 -> RECOVERING
2026-11-17T13:49:10 RECOVERING -> SECONDARY
2026-11-17T13:49:10 syncing from aen-mongo-05
2026-11-17T13:49:10 Starting rollback: OplogStartMissing: the sync source's oplog and our oplog have diverged, going into rollbac
2026-11-17T13:49:10 SECONDARY -> ROLLBACK
2026-11-17T13:49:10 Rollback common point {'$timestamp': {'t': 1794923251, 'i': 465}}
2026-11-17T13:49:11 Operations reverted by rollback: insert 201 update 0 delete 0
2026-11-17T13:49:11 Rollback complete
2026-11-17T13:49:11 ROLLBACK -> SECONDARY
2026-11-17T13:49:11 syncing from aen-mongo-05
aen-mongo-07 rolled back 201 inserts. Here’s why. I start the primary first, so aen-mongo-05 came up first, from the oldest of the three snapshots. With aen-mongo-06’s vote, it won the election and started a new term from its own, earlier point. By the time aen-mongo-07 came up and picked aen-mongo-05 as its sync source, their histories no longer matched. That’s the OplogStartMissing line: the two oplogs had diverged. aen-mongo-07 walked back to the Rollback common point, the last operation both members agreed on, and everything after it was 201 inserts that the new primary never had. Once a new primary has started a new term without them, they can’t be kept. MongoDB reverted them and wrote them to rollback files under the member’s dbPath, which is where I found them on aen-mongo-07. This is a data loss event.
The set ended up consistent, but the restore point was decided by which member won the first election, not by me, and a member had to roll back to get there. That’s the difference the lock makes. With it, every member’s snapshot is taken at the same opTime, there’s nothing for anyone to roll back, and the restore point is the one I recorded.
Why This Matters
- One point in time across three arrays: Each array’s snapshot is atomic on its own. Locking the primary is what lines them up, and the restore proved it to the document. You need the lock for consistency across multiple arrays, when you’re using an array local snapshot. Remember, if you want you can replicate from the primary to the other two arrays like we did in this post.
- The lock is short and on one member: Writes were blocked for at most 4.3 seconds, and the three snapshots were 0.9 seconds of that. Reads kept working the whole time, and the secondaries never needed a lock of their own.
- Nothing moves between arrays to restore: Every member came back from its own array’s snapshot, and every overwrite was a metadata operation, so the array’s part of the restore takes the same time whether the database is 10 GB or 10 TB. What’s left is
mongodstopping and starting. - The restore checks itself: The counts recorded under the lock, the gap-free writer sequence, and the matching
dbHashturn “it came back” into “it came back exactly.” - Without the lock, someone rolls back: Restored from local snapshots taken 452 milliseconds apart, the member that was furthest ahead rolled back 201 inserts, and an election decided the restore point, not me. With the lock, every member came back at the point I recorded, with nothing to roll back.
Wrapping Up
This is the 2025 recipe moved to a replica set, Python, and a fleet: stop writes at the source, snapshot every array, unlock, and keep the numbers you recorded under the lock so the restore can prove itself. To rebuild members, you don’t need any of this, and the reseed post shows the no-lock way. The lock earns its few seconds when you want every member back from its own array at one known moment, with nothing to roll back and a count you can check the restore against, to the document.
Two things to keep in your head when you build this yourself. First, put the unlock in a finally, always, and keep the locked window small by doing the lock, the catch-up wait, and the counts in one call. Second, make sure your test writer is really writing through the freeze, because a writer that died quietly makes any freeze look perfect. Ask me how I know. While you’re at it, check every overwrite response, since a failed overwrite comes back as an error response instead of raising. And if you’re running a sharded cluster, you’ll need to stop the balancer and lock through mongos, which is what the 2025 post covers, and we’ll dive into more detail later in this series.
MongoDB’s documentation on db.fsyncLock() and backups with file system snapshots is worth reading alongside this. Next, I’ll add a brand new member on a fourth array with Everpure Fusion. Get out in your lab and try it, and let me know how it works in your environment.
MongoDB on Everpure Series
- Why Run MongoDB on Everpure FlashArray?
- Snapshot Recovery of a MongoDB Replica Set on a Single FlashArray
- Replacing the MongoDB Replica Set Initial Sync with FlashArray Snapshots
- When You Need fsyncLock: Restoring a MongoDB Replica Set from Local Snapshots on Every FlashArray (This Post)
- Expanding a MongoDB Replica Set with Everpure Fusion
- Where FlashBlade Fits When MongoDB Lives on FlashArray
- A Tag-Based Snapshot Catalog for a MongoDB Replica Set Across FlashArrays with Everpure Fusion
Related: App-Consistent MongoDB Snapshots Across Multiple Pure Storage FlashArrays, my earlier post on snapshotting a sharded MongoDB cluster across two arrays.