Expanding a MongoDB Replica Set with Everpure Fusion
Page content
So far in this series I’ve been rebuilding members on arrays I already had. In my reseed post, I rebuilt aen-mongo-06 from a replicated snapshot of the primary. This time I wanted to grow the replica set onto a fourth array. That’s usually where you open a storage ticket, wait for someone to pick an array and carve some volumes, and then start an initial sync that copies every document over the network. Two waits, and the second one gets longer every time your database grows.
In this post, I’m going to show you how I describe a MongoDB member as an Everpure Fusion workload preset, ask Fusion where it should go, and deploy it. Then I’ll bring it into the replica set from a replicated snapshot of the primary. Fusion presets and placement show up here alongside MongoDB elections and votes, so if either side is new to you, the terms list in the intro has them in plain words. Let’s go.
The Problem We’re Solving
Adding a member to a replica set is really two projects stacked on top of each other:
- The storage project: Someone decides which array has room, someone carves the volumes, someone remembers the naming convention and the protection group, and someone else presents the volumes to a host. Every one of those steps is a place for a mistake, and none of it is the interesting part of adding a replica.
- The data project: A brand new member starts with an initial sync. It copies every document from another member and rebuilds every index, and the time that takes grows with your data. The member it syncs from carries that read load the whole time.
Fusion goes after the first one. You describe what a member needs once, as a preset. When you need another member, you ask Fusion where it fits and deploy from the preset. A storage snapshot takes care of the second one, the same way it did in my reseed post.
My Lab Setup
rs-multi has three members on three arrays. The new member is aen-mongo-04, a VM on the same ESXi cluster, with nothing under /u01 yet. Its array is flasharray04, the fourth array in the fleet.
aen-mongo-05 aen-mongo-06 aen-mongo-07 aen-mongo-04
PRIMARY SECONDARY SECONDARY (new)
flasharray01 flasharray02 flasharray03 flasharray04
│ ▲
└────────────────── replication, seed from here ──────────────────┘
Every member uses the same layout, with the journal and the local database symlinked out of the dbPath, as in the snapshot recovery post:
| Mount | Volume | In aen-mongo-pg-multi |
|---|---|---|
/u01/data |
aen-mongo-multi-0N-data |
Yes |
/u01/journal |
aen-mongo-multi-0N-journal |
Yes |
/u01/oplog |
aen-mongo-multi-0N-oplog |
Yes |
/u01/log |
aen-mongo-multi-0N-log |
No |
So a new member is four volumes, and three of them go in the protection group. That’s exactly the kind of thing I don’t want to remember by hand every time.
To seed the new member from snapshot, I need a snapshot of an existing member on flasharray04. Any member works, primary or secondary, as long as its protection group replicates to flasharray04. I use the primary, aen-mongo-05, the same seed as in the reseed post. flasharray01, the primary’s array, and flasharray04 are joined by a sync-replication (ActiveCluster) connection, and that kind of connection can also carry protection group replication too. So the only change is adding flasharray04 as a replication target of the primary’s protection group. The reseed post showed that a member’s snapshot is crash-consistent on its own when its data, journal, and oplog are in one protection group, so there’s no fsyncLock here either. Here only the new member comes from a snapshot, and the rest of the set stays live.
What You’ll Need
Here’s what I’m running, and what you’ll need to follow along:
- Four FlashArrays: One per member, including the new member’s array.
- An Everpure Fusion fleet: The new member’s array has to be in the same fleet as the others. I connect to the fleet’s gateway array, and that connection is
fain the code. The array calls are routed withcontext_names. - A replication connection from the seed member’s array to the new array: Here that’s the sync-replication connection between
flasharray01andflasharray04. The sync connection can carry the async replication connection too. - A protection group with the same name on the new array, for the new member’s volumes, so one group name covers every member, and the new array as a replication target of the seed member’s protection group:
fa.post_protection_groups(names=["aen-mongo-pg-multi"], context_names=["flasharray04"])
fa.post_protection_groups_targets(group_names=["aen-mongo-pg-multi"], member_names=["flasharray04"],
context_names=["flasharray01"])
- A host group on the new array for your hosts: Mine is
vc01-Workload-Cluster-1, for the ESXi cluster the VMs run on. - A Linux host for the new member: RHEL 9.6 in my lab, with MongoDB Enterprise 8.0.21 and
mongosh2.8.1, the same as the other members. - A control machine: Python 3 with
py-pure-client(1.99.0 in my lab), SSH key access to each host as a user with passwordless sudo, and array credentials inFA_ENDPOINT,FA_USERNAME, andFA_PASSWORD. - A way to present volumes to the host: My VMs get their FlashArray volumes as physical RDMs, so I present to the host. On bare metal, that’s your normal host connection and rescan.
Step 1: Describe a Member as a Preset
A Fusion workload preset is a template for a set of storage resources. If I describe what one MongoDB member needs once, then adding the fifth or the tenth member is a deploy, not a design exercise. This is super important in a system like Mongo where you can have many nodes/members in a cluster. Fusion presets provide consistency in deployment for all the nodes/members in the cluster in terms of storage topology and protection policies.
For my lab, a member is a 3 TiB data volume and 100 GiB each for the journal, oplog, and log. The data volume matches the size of the other members’ data volumes, because I’m about to overwrite it with a copy of one. The preset also sets a naming pattern and an app=mongodb tag, and it names the flasharray-c storage class, which matters in Step 2. And it gives every member local snapshots from the moment it’s deployed: one every 10 minutes, kept for a day.
size = {"data": 3 * 2**40, "journal": 100 * 2**30, "oplog": 100 * 2**30, "log": 100 * 2**30}
preset = PresetWorkload(
name="aen-mongo-member",
description="One MongoDB replica set member: data, journal, oplog and log",
workload_type="Custom",
platform_features=["fa_block"],
placement_configurations=[{"name": "member-placement",
"storage_class": {"name": "flasharray-c", "resource_type": "storage-classes"}}],
# A local snapshot every 10 minutes, kept for a day, in a protection group named after the workload
snapshot_configurations=[{"name": "snap-10min", "policy_action": "always_create",
"naming_patterns": [{"template": "{{workload.name}}-snap-10min"}],
"rules": [{"every": str(10 * 60 * 1000), "keep_for": str(24 * 3600 * 1000)}]}],
volume_configurations=[{"name": role, "count": "1", "provisioned_size": str(size[role]),
"placement_configurations": ["member-placement"],
# data, journal, and oplog share the policy; the log gets none
"snapshot_configurations": ["snap-10min"] if role != "log" else [],
"naming_patterns": [{"template": f"{{{{workload.name}}}}-{role}"}]}
for role in ["data", "journal", "oplog", "log"]],
workload_tags=[{"namespace": "default", "key": "app", "value": "mongodb", "copyable": "true"}])
r = fa.put_presets_workload(names=["aen-mongo-member"], preset=preset, context_names=["fleet01"])
# prints the preset's revision, each volume configuration, and the snapshot policy from r, shown below
Output:
Updated preset fleet01:aen-mongo-member, revision 4
data 3072 GiB {{workload.name}}-data snapshots: snap-10min
journal 100 GiB {{workload.name}}-journal snapshots: snap-10min
oplog 100 GiB {{workload.name}}-oplog snapshots: snap-10min
log 100 GiB {{workload.name}}-log snapshots: -
snap-10min: every 10 min, keep 24 h, always_create, named {{workload.name}}-snap-10min
Look at the revision. This preset is on its fourth version: I went through two earlier layouts while I worked this out, and then I added the snapshot policy. put_presets_workload is a full replace, not a patch, so it swapped the whole preset for the four-volume version, and you send the complete definition every time. Below that, each volume configuration shows its size, its naming template, and its snapshot policy, and the last line is the policy itself. When the member layout changes, I change it here, in one place. Workloads you’ve already deployed keep the revision they were created from, and only new deploys pick up the change. A few more things about presets worth knowing:
- Presets belong to the fleet: My first attempt used an array as the context, and it failed with
Creating a preset in the array context is not supported.The fleet is the context, which is why the call above passesfleet01. - One placement per preset: The preset schema allows exactly one placement configuration, and everything tied to it lands on the same array. For a MongoDB member, that’s exactly what I want. A member’s data, journal, and oplog have to be on one array to share a protection group, and that’s what makes its snapshot crash-consistent.
- The name pattern: The workload name becomes the prefix, so a workload named
aen-mongo-multi-04getsaen-mongo-multi-04-data,-journal,-oplog, and-log, matching the other members. - The snapshot policy: Times are in milliseconds, so
everyis 10 minutes andkeep_foris 24 hours, which is 144 snapshots per member. The data, journal, and oplog share the policy, so each scheduled snapshot captures all three at the same instant, and that’s what makes it crash-consistent. The log gets no policy, the same as inaen-mongo-pg-multi.always_creategives every workload its own protection group, and the naming pattern puts the workload’s name on it, so two members never share a schedule. - Why a schedule: A member’s own snapshot can rebuild that member only while the local snapshot is inside the oplog window. An older one is too far behind to catch up, as the reseed post showed. A snapshot every 10 minutes keeps a fresh one on each member’s array. Otherwise, you will need to replicate a snapshot off the primary.
- What a schedule doesn’t do: Each member’s schedule runs on its own array, so the members’ snapshots are never taken at the same moment. To restore every member from its own local array’s snapshot to one point in time, you still need
fsyncLock, as in my fsyncLock post.
Step 2: Ask Fusion Where It Should Go
Next, I ask Fusion for a placement recommendation. I add a constraint so it only considers arrays that have the host group my ESXi cluster uses, because there’s no point recommending an array the VM can’t reach. Fusion computes the recommendation asynchronously, so I poll it until it’s done.
ask = WorkloadPlacementRecommendation(
additional_constraints={"required_resource_references": [
{"allowed_values": [{"resource_type": "host-groups", "name": "vc01-Workload-Cluster-1"}]}]},
results_limit=5)
r = fa.post_workloads_placement_recommendations(inputs=ask, preset_names=["fleet01:aen-mongo-member"])
name = list(r.items)[0].name
while True:
rec = list(fa.get_workloads_placement_recommendations(names=[name]).items)[0]
if rec.status != "processing":
break
time.sleep(3)
print("status:", rec.status)
for res in rec.results or []:
print(res.score, [t.name for t in res.targets])
Output:
status: completed
optimal ['flasharray04']
Fusion’s answer is flasharray04, scored optimal, and it’s the only result, even though I asked for up to five. That’s the answer I expected. My preset names the flasharray-c storage class, and flasharray04 is the only C-series array in my fleet, so the storage class did the narrowing and the host group constraint confirmed the VM can reach it. In a bigger fleet, this is where the workflow pays off: the preset says what a member needs, Fusion says where it fits, and nobody opens a spreadsheet of free capacity.
Step 3: Deploy the Workload
With the array chosen, I deploy the workload from the preset on that array and wait for its four volumes to show up.
fa.post_workloads(names=["aen-mongo-multi-04"], workload=WorkloadPost(),
preset_names=["fleet01:aen-mongo-member"], context_names=["flasharray04"])
print("Deployed workload aen-mongo-multi-04 on flasharray04")
# Wait until all four volumes are listed
roles = ["data", "journal", "oplog", "log"]
while len(vols := list(fa.get_volumes(filter="name='aen-mongo-multi-04-*'", destroyed=False,
context_names=["flasharray04"]).items)) < len(roles):
time.sleep(3)
# prints each volume's name, size, and serial in role order, shown in the output below
The preset gave me four volumes with the names and sizes I asked for, and the workload carries the app=mongodb tag. Fusion also built the snapshot policy as a new protection group that belongs to the workload, aen-mongo-multi-04-snap-10min, with the data, journal, and oplog in it and the schedule already on. That covers local snapshots on flasharray04. It’s not the whole job, though.
The replication and restores in this series run through aen-mongo-pg-multi, the group every other member’s array already has, so one group name covers the whole replica set. The preset doesn’t add volumes to a group it didn’t create, and it has no section for host connections, so I do both steps with the API. That leaves each member’s data, journal, and oplog in two groups: the workload’s own group for the 10-minute local snapshots, and aen-mongo-pg-multi for everything else. Only three of the four volumes go in aen-mongo-pg-multi, because the log stays out:
fa.post_protection_groups_volumes(group_names=["aen-mongo-pg-multi"], context_names=["flasharray04"],
member_names=[f"aen-mongo-multi-04-{role}" for role in ["data", "journal", "oplog"]])
fa.post_connections(host_group_names=["vc01-Workload-Cluster-1"], context_names=["flasharray04"],
volume_names=[f"aen-mongo-multi-04-{role}" for role in ["data", "journal", "oplog", "log"]])
# prints both groups' members, the snapshot schedule, and the connection, shown in the output below
In my lab, the next part is presenting the four volumes to the aen-mongo-04 VM as physical RDMs through vCenter. That’s specific to my environment. On bare metal or another hypervisor, it’s whatever you normally do to get a LUN in front of a host. After that, the only volume that gets a new file system is the log. The other three are about to be overwritten with copies of the primary’s, so formatting them would be wasted work.
Once the disks are presented, I rescan the SCSI hosts on the VM, format and mount the log volume by its device path, and list what the guest sees. ssh() is a small helper that runs a command on a host over SSH and returns what it printed. Its partner mongosh(), which shows up in the next step, does the same for a piece of JavaScript run through mongosh, with an optional connection string.
ssh("aen-mongo-04", 'for h in /sys/class/scsi_host/host*; do echo "- - -" | sudo tee $h/scan >/dev/null; done; sleep 3')
# The guest sees each volume at /dev/disk/by-id/wwn-0x624a9370<serial>
serial = {v.name.rsplit("-", 1)[1]: v.serial for v in vols}
log = f"/dev/disk/by-id/wwn-0x624a9370{serial['log'].lower()}"
ssh("aen-mongo-04", f"sudo mkfs.xfs -f -q {log} && uuid=$(sudo blkid -s UUID -o value {log}) && "
f"echo \"UUID=$uuid /u01/log xfs defaults,nofail 0 2\" | sudo tee -a /etc/fstab >/dev/null && "
"sudo mkdir -p /u01/log && sudo mount /u01/log && sudo chown mongod:mongod /u01/log")
print(ssh("aen-mongo-04", "lsblk -dno NAME,SIZE,WWN | grep 624a9370; findmnt -rn -o TARGET,SOURCE,SIZE | grep ^/u01"))
Here’s what Step 3 printed, from the deploy to the guest’s view of its disks:
Output:
Deployed workload aen-mongo-multi-04 on flasharray04
aen-mongo-multi-04-data 3072 GiB F269A91400C5408F00B34BAE
aen-mongo-multi-04-journal 100 GiB F269A91400C5408F00B34BAF
aen-mongo-multi-04-oplog 100 GiB F269A91400C5408F00B34BB0
aen-mongo-multi-04-log 100 GiB F269A91400C5408F00B34BB1
aen-mongo-pg-multi on flasharray04 ['aen-mongo-multi-04-data', 'aen-mongo-multi-04-journal', 'aen-mongo-multi-04-oplog']
aen-mongo-multi-04-snap-10min on flasharray04 ['aen-mongo-multi-04-data', 'aen-mongo-multi-04-journal', 'aen-mongo-multi-04-oplog'] snapshot every 10 min, enabled True, keep all for 24 h
Connected all four to vc01-Workload-Cluster-1
...output omitted...
sdb 3T 0x624a9370f269a91400c5408f00b34bae
sdd 100G 0x624a9370f269a91400c5408f00b34baf
sdg 100G 0x624a9370f269a91400c5408f00b34bb0
sdh 100G 0x624a9370f269a91400c5408f00b34bb1
/u01/log /dev/sdh 99.9G
I trimmed the vCenter attach lines and the node’s older disks out of the listing. Match the serials at the top with the WWNs at the bottom. Each FlashArray volume’s serial is the tail end of the WWN the guest sees, right after the 624a9370 prefix, so the log volume, F269A91400C5408F00B34BB1, is sdh in the guest, and that’s the one mounted at /u01/log. That’s why I build device paths from the serial instead of guessing at sdb and sdd. The new disks didn’t even land on consecutive device letters, because the node’s older disks sit between them. At this point the member has its four volumes, the right three are in both protection groups, and only the log is mounted. The schedule doesn’t wait for MongoDB, either: aen-mongo-multi-04-snap-10min.1 was taken at the first 10-minute mark after the deploy.
Step 4: Seed from a Snapshot of the Primary
The new data, journal, and oplog volumes are empty. In my reseed post, I rebuilt a member from a replicated snapshot, and this is the same move with a new destination. I snapshot the primary’s protection group on flasharray01 with replicate_now and no fsyncLock, then write 50,000 more orders on the primary. Those writes aren’t in the snapshot, so the new member has to get them from the oplog. PRIMARY is aen-mongo-05, and URI is the replica set connection string for aen-mongo-05, aen-mongo-06, and aen-mongo-07.
since = ssh("aen-mongo-04", "date -u +%Y-%m-%dT%H:%M:%S") # for reading aen-mongo-04's log at the end
t0 = time.time()
snap = fa.post_protection_group_snapshots(source_names=["aen-mongo-pg-multi"], context_names=["flasharray01"],
replicate_now=True,
protection_group_snapshot=ProtectionGroupSnapshotPost(suffix="add04"))
name = list(snap.items)[0].name # aen-mongo-pg-multi.add04
print(f"Snapshot: {name}, no fsyncLock, {time.time() - t0:.1f} seconds for the snapshot call")
# Writes after the snapshot: aen-mongo-04 has to pick these up from the oplog
mongosh(PRIMARY, "db.getSiblingDB('demo').orders.insertMany(Array.from({length: 50000}, "
"(_, i) => ({ _id: 32000000 + i, note: 'after the snapshot' })))", uri=URI)
print("Wrote 50,000 orders on the primary after the snapshot")
The primary’s group replicates to all three of the other arrays, so I wait for the copy on flasharray04 specifically. When it’s there, I overwrite the new member’s three volumes from aen-mongo-05’s, matched by role, and mount them by device path. An overwrite replaces a volume’s contents, not the volume, so the serials and device paths from Step 3 still apply. The copies carry aen-mongo-05’s XFS UUIDs, so I mount by device the same way I did in the reseed post.
# Wait for the replicated snapshot to finish its transfer to flasharray04
while not getattr(next(iter(fa.get_protection_group_snapshots_transfer(filter=f"name='flasharray01:{name}'",
context_names=["flasharray04"]).items), None),
"completed", None):
time.sleep(2)
for role in ["data", "journal", "oplog"]:
src = f"flasharray01:{name}.aen-mongo-multi-05-{role}"
fa.post_volumes(names=[f"aen-mongo-multi-04-{role}"], context_names=["flasharray04"], overwrite=True,
volume=VolumePost(source=Reference(name=src)))
# Same volumes as in Step 3, so the same serials and device paths
for role in ["data", "journal", "oplog"]:
ssh("aen-mongo-04", f"sudo mkdir -p /u01/{role} && "
f"sudo mount /dev/disk/by-id/wwn-0x624a9370{serial[role].lower()} /u01/{role}")
Along the way the script prints the transfer time and one line per overwrite. You’ll find those in the combined output after Step 5.
Step 5: Add the Member
Now I start mongod on aen-mongo-04 with the same flags as every other member, including --directoryperdb. BIN is the directory that holds the MongoDB 8.0.21 binaries. Just before the start, the script runs the same symlink step as in the snapshot recovery post, creating dbPath/journal and dbPath/local only if they aren’t already there. On a copy of aen-mongo-05’s data volume, they already are. Then I add the member from the primary.
DBPATH = "/u01/data/rs-multi"
ssh("aen-mongo-04", f"sudo runuser -u mongod -- {BIN}/mongod --replSet rs-multi --bind_ip_all --dbpath {DBPATH} "
f"--directoryperdb --logpath /u01/log/mongod.log --logappend --oplogSize 990 "
f"--wiredTigerCacheSizeGB 2 --fork")
mongosh(PRIMARY, "rs.add('aen-mongo-04.fsa.lab:27017')", uri=URI)
Then I wait for aen-mongo-04 to reach SECONDARY and for every member to reach the same last operation. After 20 more seconds, I check the counts and the dbHash of the demo database on all four members. The helpers here are thin wrappers around mongosh:
state(): Reads one member’sstateStrfromrs.status()on the primary.wait_until_caught_up(): Pollsrs.status()up to 60 times, 2 seconds apart, until every member reports the same optime, and returns whether they got there.members(): Prints each member’s name and state.count()anddbhash(): Connect directly to one member and return thedemo.ordersdocument count and themd5fromdbHashondemo.transitions(): Readsaen-mongo-04’smongodlog on/u01/logand prints its replica set state changes, and a few key messages, sincesince.
while state("aen-mongo-04") != "SECONDARY": # that member's state from rs.status()
time.sleep(2)
print(f"aen-mongo-04 is SECONDARY {time.time() - t0:.0f} seconds after the snapshot")
caught_up = wait_until_caught_up() # wait until every member reports the same optime
time.sleep(20)
print(members()) # each member's name and state from rs.status()
nodes = ["aen-mongo-04", "aen-mongo-05", "aen-mongo-06", "aen-mongo-07"]
print("orders: ", " ".join(f"{n[-2:]}: {count(n)}" for n in nodes)) # orders documents, read on that member
# md5 of dbHash on demo, read on that member
print("demo dbHash:", " ".join(f"{n[-2:]}: {dbhash(n)[:12]}" for n in nodes))
print("all members on the same last operation:", caught_up)
print(transitions("aen-mongo-04", since)) # state changes from the member's log since 'since'
Here’s everything Steps 4 and 5 printed, from the snapshot to the checks:
Output:
Snapshot: aen-mongo-pg-multi.add04, no fsyncLock, 0.4 seconds for the snapshot call
Wrote 50,000 orders on the primary after the snapshot
Replicated to flasharray04 in 7 seconds
Overwrote aen-mongo-multi-04-data from flasharray01:aen-mongo-pg-multi.add04.aen-mongo-multi-05-data
Overwrote aen-mongo-multi-04-journal from flasharray01:aen-mongo-pg-multi.add04.aen-mongo-multi-05-journal
Overwrote aen-mongo-multi-04-oplog from flasharray01:aen-mongo-pg-multi.add04.aen-mongo-multi-05-oplog
...output omitted...
aen-mongo-04 is SECONDARY 20 seconds after the snapshot
aen-mongo-05.fsa.lab:27017 PRIMARY
aen-mongo-06.fsa.lab:27017 SECONDARY
aen-mongo-07.fsa.lab:27017 SECONDARY
aen-mongo-04.fsa.lab:27017 SECONDARY
...output omitted...
orders: 04: 1230502 05: 1230502 06: 1230502 07: 1230502
demo dbHash: 04: ac330a658e20 05: ac330a658e20 06: ac330a658e20 07: ac330a658e20
all members on the same last operation: True
2026-11-17T21:10:32 not in the replica set config yet
2026-11-17T21:10:32 STARTUP -> REMOVED
2026-11-17T21:10:33 REMOVED -> STARTUP2
2026-11-17T21:10:33 No initial sync required. Attempting to begin steady replica
2026-11-17T21:10:33 STARTUP2 -> RECOVERING
2026-11-17T21:10:33 RECOVERING -> SECONDARY
2026-11-17T21:10:33 syncing from aen-mongo-05
aen-mongo-04 was a secondary 20 seconds after the snapshot. The replicated snapshot was ready on flasharray04 7 seconds after the snapshot call started, and the 50,000-order write ran inside that window. Look at the orders line: all four members hold the same 1,230,502 orders, including the 50,000 written after the snapshot. The dbHash line (I print the first 12 characters) reads ac330a658e20 on all four, so the contents match too, and the last line says every member ended on the same optime. That’s a brand new member on a brand new array with the same data as the rest of the set, and not one document crossed the network in an initial sync.
The log on aen-mongo-04 is worth reading line by line. It starts in REMOVED, and that’s expected. The copy carries aen-mongo-05’s replica set configuration, which lists aen-mongo-05, aen-mongo-06, and aen-mongo-07 but not aen-mongo-04, so mongod can’t find itself in it. Crash recovery and replication recovery on the copied files already happened when mongod started, before it noticed it wasn’t in the config. A second later the primary’s rs.add reaches it with a configuration that does include it.
From there it finishes the same path as in my reseed post: No initial sync required, and then syncing from aen-mongo-05 to pick up the 50,000 orders it missed.
A Note on Votes
rs.add with just a host name adds a full voting member with priority 1. The script prints the new member’s entry from rs.conf(), which I trimmed from the output, and it shows votes: 1 and priority: 1. So rs-multi now has four voting members. MongoDB allows that, but its documentation recommends an odd number of voting members, and a fourth vote buys you nothing on availability. A majority of four is three, so the set still survives only one member failure, the same as with three members, and a majority write now needs one more acknowledgment. In my lab, I left it that way. In production, I’d either add a fifth member from the same preset, which is now one deploy away, or add this one with votes: 0, priority: 0 until the fifth one exists.
Why This Matters
- The member is described once: The sizes, names, tag, and snapshot policy live in one preset. When the layout changed, and when I added the 10-minute snapshots, I changed the preset, not a runbook.
- Placement keeps a member’s volumes together: All four volumes land on the same array. That’s what lets the data, journal, and oplog share a protection group, and it’s what makes a snapshot of the member crash-consistent.
- The member comes in from a snapshot, not an initial sync: The new node was a secondary 20 seconds after the snapshot, with a
dbHashthat matched the rest of the set. An initial sync copies every document and rebuilds every index, so it grows with your data. The snapshot and the overwrite take the same time whether the database is 10 GB or 10 TB. The part that does move data is replication to the new array. The first transfer to an array sends all of the volumes’ data, and after that only what changed since the last replicated snapshot, so plan the first one around your data size and the rest around your change rate. - Your replication targets choose your seed source: You can seed from any member, primary or secondary, whose protection group replicates to the new array. I added
flasharray04as a target of the primary’s group, and a secondary’s group would work the same way. Check which groups target which arrays before you need it.
Wrapping Up
Growing the replica set onto another array came down to four moves: describe the member, ask Fusion where it fits, deploy it, and seed it from a snapshot of a member that was already running. The new member joined as a full secondary, with the same data as the other three, 20 seconds after the snapshot.
Two things to keep in your head if you try this. First, this preset makes volumes and their own snapshot schedule. It doesn’t add them to your existing protection group or connect them to hosts, so put those steps right after the deploy where you can’t forget them. Second, a seeded member wakes up with its source’s replica set configuration, so expect REMOVED until rs.add reaches it, and think about the vote count before you add a fourth member. MongoDB’s documentation on adding members to a replica set is worth reading alongside this. Get out in your lab and try it, and let me know how it works in your environment.
MongoDB on Everpure Series
- Why Run MongoDB on Everpure FlashArray?
- Snapshot Recovery of a MongoDB Replica Set on a Single FlashArray
- Replacing the MongoDB Replica Set Initial Sync with FlashArray Snapshots
- When You Need fsyncLock: Restoring a MongoDB Replica Set from Local Snapshots on Every FlashArray
- Expanding a MongoDB Replica Set with Everpure Fusion (This Post)
- Where FlashBlade Fits When MongoDB Lives on FlashArray
Related: App-Consistent MongoDB Snapshots Across Multiple Pure Storage FlashArrays, my earlier post on snapshotting a sharded MongoDB cluster across two arrays.