App-Consistent MongoDB Snapshots Across Multiple Pure Storage FlashArrays

Introduction

When you’re running MongoDB at scale with data distributed across multiple Pure Storage FlashArrays, achieving truly consistent backups becomes a critical and interesting technical challenge. In this post, I’m walking through an automated snapshot and recovery solution for a sharded MongoDB cluster running across two separate FlashArrays.

Important: This demonstration uses a sharded cluster with data distributed across multiple shard servers, not a replica set. Sharded deployments introduce additional complexity for backup operations because data is partitioned across multiple servers and storage arrays. The coordination requirements are more demanding; we must ensure consistency across config servers, multiple shards, and the mongos routers simultaneously.

The key challenge? We need app-consistent snapshots across physically separate storage systems, all while maintaining data integrity in a distributed database environment. And we need to do this with minimal impact on production workloads.

This blog post shows how combining MongoDB’s fsyncLock() mechanism with Pure Storage’s protection group snapshots creates a point-in-time recovery solution that completes in just over 5 seconds while the database remains online and consistent across both arrays.

Watch the complete walkthrough: MongoDB Multi-Array Snapshot Demo on YouTube

The Value Proposition for Large Shared Systems

Before diving into the technical details, let’s talk about why this matters, especially for very large shared MongoDB deployments.

The Challenge with Traditional Backups

When MongoDB clusters grow to multi-terabyte scale and span multiple storage arrays, traditional backup approaches start showing their limitations:

  1. Time: Logical backups with mongodump can take hours or even days on large datasets
  2. Network Impact: Streaming terabytes across the network for backup puts pressure on production infrastructure
  3. Consistency: Coordinating consistent backups across multiple shards and arrays requires careful orchestration
  4. Recovery Time Objective (RTO): Restoring from logical backups is equally time-consuming
  5. Storage Efficiency: Traditional backups often can’t leverage deduplication and compression effectively

The Storage Snapshot Advantage

Storage-level snapshots, particularly Pure Storage’s copy-on-write implementation, fundamentally change this equation:

  • Speed: Snapshots complete in seconds regardless of dataset size (our demo: 5 seconds)
  • Space Efficiency: Pure Storage’s data reduction (4x on one array, 4.5x on another in our demo) means snapshots consume minimal additional space
  • Instant Restore: Restoring is metadata-based, not a data copy operation
  • No Network Impact: Snapshots happen at the storage layer without database network traffic
  • Application Consistency: When coordinated with fsyncLock(), we get app-consistent snapshots across distributed systems

For organizations running mission-critical MongoDB workloads on shared infrastructure supporting multiple applications and teams, this approach delivers enterprise-grade data protection with minimal operational overhead.

The Architecture

This environment consists of:

  • MongoDB Cluster: 2-server sharded deployment
    • Server 1 (192.168.1.8): Config server, Shard 1, and mongos router
    • Server 2 (192.168.1.9): Shard 2
  • Storage: Two Pure Storage FlashArrays
    • flasharray1 hosting Server 1’s data
    • flasharray2 hosting Server 2’s data
  • Protection Groups: pg-aen-mongo-01 and pg-aen-mongo-02
  • Data Mount Point: /data on both servers

The sharding configuration distributes data across both arrays, making coordinated snapshots essential for consistency.

Prerequisites and Important Notes

Before implementing this solution, ensure you have:

  1. MongoDB 7.1+ (or 7.0.2+, 6.0.11+, 5.0.22+) - required for fsyncLock() to work on mongos in sharded clusters
  2. MongoDB shell (mongosh) installed and accessible
  3. PureStoragePowerShellSDK2 module installed
  4. Appropriate credentials for FlashArray management
  5. PowerShell remoting over SSH enabled on MongoDB servers
  6. MongoDB data directories on Pure Storage volumes configured in protection groups

Important: This script is designed to be executed section by section, not as a single automated run. Each step includes validation and user prompts to prevent accidental data loss.

The Script Walkthrough

Let’s walk through each step with actual execution output from my lab environment.

Step 0: Initial Setup and Verification

The script begins by establishing connectivity to both FlashArrays and creating PowerShell remote sessions to the MongoDB servers:

# Initialize variables for MongoDB servers and FlashArrays
$MongoServer1     = '192.168.1.8'                                              # MongoDB Server 1 (shard1)
$MongoServer2     = '192.168.1.9'                                              # MongoDB Server 2 (shard2)
$SSHUser          = 'nocentino'                                                # SSH username for MongoDB servers
$ArrayName1       = 'flasharray1.puretec.purestorage.com'                      # First FlashArray
$ArrayName2       = 'flasharray2.puretec.purestorage.com'                      # Second FlashArray
$PGroupName1      = 'pg-aen-mongo-01'                                          # Protection Group on FlashArray1
$PGroupName2      = 'pg-aen-mongo-02'                                          # Protection Group on FlashArray2
$FlashArray1DbVol = 'vvol-aen-mongo-01-2a764477-vg/Data-8a047da9'              # Volume name on FlashArray1
$FlashArray2DbVol = 'vvol-aen-mongo-02-145f0b6e-vg/Data-a04d05fb'              # Volume name on FlashArray2
$DataMountPoint   = '/data'                                                    # MongoDB data mount point

# Connect to the FlashArrays' REST APIs
Write-Host "Connecting to FlashArrays..." -ForegroundColor Cyan
$Credential = Import-CliXml -Path "$HOME\FA_Cred.xml"
$FlashArray1 = Connect-Pfa2Array -EndPoint $ArrayName1 -Credential $Credential -IgnoreCertificateError
$FlashArray2 = Connect-Pfa2Array -EndPoint $ArrayName2 -Credential $Credential -IgnoreCertificateError
Write-Host "Connected to both FlashArrays successfully!" -ForegroundColor Green

# Create PowerShell remote sessions to MongoDB servers
Write-Host "Creating remote sessions to MongoDB servers..." -ForegroundColor Cyan
$Session1 = New-PSSession -HostName $MongoServer1 -UserName $SSHUser
$Session2 = New-PSSession -HostName $MongoServer2 -UserName $SSHUser
Write-Host "Remote sessions established!" -ForegroundColor Green

Output:

Connecting to FlashArrays...
Connected to both FlashArrays successfully!
Creating remote sessions to MongoDB servers...
Remote sessions established!

Implementation Note: I’m using PowerShell remoting over SSH (New-PSSession -HostName) rather than traditional SSH command execution. This provides a more robust execution model with better error handling and the ability to run multiple commands within the same session context. Each session maintains state, which is crucial for multi-step operations.

Before taking any snapshots, we verify the current database state:

Write-Host "`n=== STEP 0: VERIFYING CURRENT DATABASE ===" -ForegroundColor Yellow
$ShowResult = Invoke-Command -Session $Session1 -ScriptBlock {
 mongosh mongodb://localhost:27017 --quiet --eval 'db.getSiblingDB("TestDB1")'
 mongosh mongodb://localhost:27017 --quiet --eval 'db.getSiblingDB("TestDB1").getCollectionNames()'
}
Write-Host "Current database and collections: $ShowResult" -ForegroundColor Green

Output:

=== STEP 0: VERIFYING CURRENT DATABASE ===
Current database and collections: TestDB1 [ 'orders', 'products', 'customers' ]

This confirms our test database TestDB1 contains three collections that are sharded across our two-server cluster. These collections will be the subject of our snapshot and recovery demonstration.

Step 1: Freezing MongoDB with fsyncLock

This is the critical step for achieving app consistency. MongoDB’s fsyncLock() command forces all pending writes to disk and blocks new write operations across the entire sharded cluster:

Write-Host "`n=== STEP 1: Freezing MongoDB with fsyncLock ===" -ForegroundColor Yellow

$Start = Get-Date

# Stop balancer and verify
$LockResult = Invoke-Command -Session $Session1 -ScriptBlock { 
 mongosh mongodb://localhost:27017 --quiet --eval 'sh.setBalancerState(false); sh.getBalancerState()'
}
if ($LockResult -match 'true') { 
    throw "Balancer did not stop." 
} else { 
 Write-Host "Balancer stopped: $LockResult (false means stopped)" -ForegroundColor Green 
}

# Lock the sharded cluster via mongos
$LockResult = Invoke-Command -Session $Session1 -ScriptBlock { 
 mongosh mongodb://localhost:27017 --quiet --eval 'db.fsyncLock()' 
}
if (($LockResult | Out-String) -notlike '*ok: 1*') { 
    throw "fsyncLock failed: $LockResult" 
} else {  
 Write-Host ($LockResult | ConvertTo-Json -Depth 10) -ForegroundColor Cyan
 Write-Host "MongoDB locked successfully" -ForegroundColor Green
}

Output:

=== STEP 1: Freezing MongoDB with fsyncLock ===
Balancer stopped: false (false means stopped)
[
 "{",
 "  numFiles: 1,",
 "  all: {",
 "    raw: {",
 "      'shard2ReplSet/192.168.1.9:27021': {",
 "        info: 'now locked against writes, use db.fsyncUnlock() to unlock',",
 "        lockCount: Long('1'),",
 "        seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',",
 "        ok: 1",
 "      },",
 "      'shard1ReplSet/192.168.1.8:27020': {",
 "        info: 'now locked against writes, use db.fsyncUnlock() to unlock',",
 "        lockCount: Long('1'),",
 "        seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',",
 "        ok: 1",
 "      },",
 "      'configReplSet/192.168.1.8:27019': {",
 "        info: 'now locked against writes, use db.fsyncUnlock() to unlock',",
 "        lockCount: Long('1'),",
 "        seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',",
 "        ok: 1",
 "      }",
 "    }",
 "  },",
 "  ok: 1,",
 "  '$clusterTime': {",
 "    clusterTime: Timestamp({ t: 1763914816, i: 4 }),",
 "    signature: {",
 "      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),",
 "      keyId: Long('0')",
 "    }",
 "  },",
 "  operationTime: Timestamp({ t: 1763914816, i: 4 })",
 "}"
]
MongoDB locked successfully

What’s Happening Here: The output is particularly important. Notice that the lock operation returns status for all three replica sets in our cluster:

  • configReplSet (config server on 192.168.1.8:27019)
  • shard1ReplSet (shard 1 on 192.168.1.8:27020)
  • shard2ReplSet (shard 2 on 192.168.1.9:27021)

Each confirms lockCount: 1, meaning writes are blocked across the entire sharded cluster. This is essential; we need all shards frozen simultaneously to ensure a consistent snapshot across both storage arrays.

Critical Protection Group Requirement: When you execute fsyncLock() on mongos (as shown above), MongoDB automatically locks all data nodes, config servers, and shards across the cluster. This means all volumes containing any MongoDB component (config servers, shards, mongos) must be in the same protection group on their respective arrays. In our environment, pg-aen-mongo-01 contains the config server and shard 1 volumes, while pg-aen-mongo-02 contains shard 2 volumes. We snapshot both protection groups immediately after the single fsyncLock() command, while writes remain frozen to maintain consistency.

Before taking any snapshots, we also stop the MongoDB balancer. The balancer is responsible for moving chunks between shards to maintain even data distribution. During snapshot operations, we don’t want chunks migrating between shards, as this could lead to inconsistent snapshots.

Step 2: Taking Snapshots on Both FlashArrays

With MongoDB frozen in a consistent state, we can now safely snapshot both FlashArrays:

Write-Host "`n=== STEP 2: Taking snapshots on both FlashArrays ===" -ForegroundColor Yellow

$SnapshotTag = "mongo-" + (Get-Date).ToString('yyyyMMdd-HHmmss-UTC')

# Take snapshot on FlashArray1
Write-Host "Creating snapshot on $ArrayName1..." -ForegroundColor Cyan
$SnapshotFlashArray1 = New-Pfa2ProtectionGroupSnapshot -Array $FlashArray1 -SourceName $PGroupName1 -Suffix $SnapshotTag
Write-Host "Snapshot created: $($SnapshotFlashArray1.Name)" -ForegroundColor Green

# Take snapshot on FlashArray2
Write-Host "Creating snapshot on $ArrayName2..." -ForegroundColor Cyan
$SnapshotFlashArray2 = New-Pfa2ProtectionGroupSnapshot -Array $FlashArray2 -SourceName $PGroupName2 -Suffix $SnapshotTag
Write-Host "Snapshot created: $($SnapshotFlashArray2.Name)" -ForegroundColor Green

Output:

=== STEP 2: Taking snapshots on both FlashArrays ===
Creating snapshot on flasharray1.puretec.purestorage.com...
Snapshot created: pg-aen-mongo-01.mongo-20251123-102017-UTC
Creating snapshot on flasharray2.puretec.purestorage.com...
Snapshot created: pg-aen-mongo-02.mongo-20251123-102017-UTC

Step 3: Unlocking MongoDB

With snapshots complete, we immediately unfreeze MongoDB and resume normal operations:

Write-Host "`n=== STEP 3: Unlocking MongoDB ===" -ForegroundColor Yellow

Write-Host "Unlocking MongoDB..." -ForegroundColor Cyan

# Always attempt to unlock all participants
$UnlockResult = Invoke-Command -Session $Session1 -ScriptBlock { 
 mongosh mongodb://localhost:27017 --quiet --eval 'db.fsyncUnlock()' 
} 
if (($UnlockResult | Out-String) -notlike '*ok: 1*') { 
 Write-Host "WARNING: fsyncUnlock may have failed: $UnlockResult" -ForegroundColor Yellow 
} else {
 Write-Host "MongoDB unlocked successfully" -ForegroundColor Green
}
    
# Resume balancer
$UnlockResult = Invoke-Command -Session $Session1 -ScriptBlock { 
 mongosh mongodb://localhost:27017 --quiet --eval 'sh.setBalancerState(true); sh.getBalancerState()'
}
if ($UnlockResult -match 'false') { 
 Write-Host "WARNING: Balancer did not resume: $UnlockResult" -ForegroundColor Yellow 
} else { 
 Write-Host "Balancer resumed: $UnlockResult (true means running)" -ForegroundColor Green 
}

$Stop = Get-Date
$Duration = ($Stop - $Start).TotalMilliseconds

Write-Host "`n=== Snapshot Complete ===" -ForegroundColor Green
Write-Host "Total snapshot time: $Duration ms" -ForegroundColor Cyan
Write-Host "Array 1 Snapshot: $($SnapshotFlashArray1.Name)" -ForegroundColor White
Write-Host "Array 2 Snapshot: $($SnapshotFlashArray2.Name)" -ForegroundColor White

Output:

=== STEP 3: Unlocking MongoDB ===
Unlocking MongoDB...
MongoDB unlocked successfully
Balancer resumed: true (true means running)

=== Snapshot Complete ===
Total snapshot time: 5092.589 ms
Array 1 Snapshot: pg-aen-mongo-01.mongo-20251123-102017-UTC
Array 2 Snapshot: pg-aen-mongo-02.mongo-20251123-102017-UTC

Performance Analysis: The entire lock-snapshot-unlock sequence completed in just 5.09 seconds. During this brief window:

  • The balancer was stopped
  • Writes were blocked across all shards (but MongoDB remained online for reads)
  • All pending writes were flushed to disk
  • Two separate storage arrays were snapshotted
  • Write capability was restored
  • The balancer was resumed
  • This can be made much faster; there are lots of remote networking calls using PowerShell remoting, as well as remote REST API calls, and also executing the snapshot calls serially, rather than in parallel. So there’s lots of room for optimization here. I’m focusing on the higher-level design rather than last-mile tuning.

For production workloads, this is an acceptable maintenance window that can run during business hours. The fsyncLock() operation blocks writes but not reads, so read-heavy workloads experience minimal impact.

We can also verify our snapshots were created successfully:

Write-Host "`n=== Recent Snapshots ===" -ForegroundColor Yellow

Write-Host "`nFlashArray1 ($ArrayName1) - Recent snapshots:" -ForegroundColor Cyan
Get-Pfa2ProtectionGroupSnapshot -Array $FlashArray1 -Name '*.mongo*' | 
 Select-Object -First 5 Name, Created | 
 Format-Table -AutoSize

Write-Host "`nFlashArray2 ($ArrayName2) - Recent snapshots:" -ForegroundColor Cyan
Get-Pfa2ProtectionGroupSnapshot -Array $FlashArray2 -Name '*.mongo*' | 
 Select-Object -First 5 Name, Created | 
 Format-Table -AutoSize

Output:

=== Recent Snapshots ===

FlashArray1 (flasharray1.puretec.purestorage.com) - Recent snapshots:

Name                                      Created
----                                      -------
pg-aen-mongo-01.mongo-20251123-100501-UTC 11/23/2025 4:05:02 PM
pg-aen-mongo-01.mongo-20251123-101414-UTC 11/23/2025 4:14:14 PM
pg-aen-mongo-01.mongo-20251123-102017-UTC 11/23/2025 4:20:18 PM


FlashArray2 (flasharray2.puretec.purestorage.com) - Recent snapshots:

Name                                      Created
----                                      -------
pg-aen-mongo-02.mongo-20251123-100501-UTC 11/23/2025 4:05:02 PM
pg-aen-mongo-02.mongo-20251123-101414-UTC 11/23/2025 4:14:15 PM
pg-aen-mongo-02.mongo-20251123-102017-UTC 11/23/2025 4:20:18 PM

Step 4: Demonstrating Database Deletion

To demonstrate that our snapshot and restore process works, we’ll intentionally delete the entire test database. This simulates a disaster recovery scenario:

Write-Host "`n=== DELETING DATABASE TO DEMONSTRATE RESTORE PROCESS ===" -ForegroundColor Red
$ConfirmDelete = Read-Host "Type 'DELETE' to drop the test database"
if ($ConfirmDelete -eq 'DELETE') {
 Write-Host "Dropping test database 'TestDB1' on MongoDB server 1..." -ForegroundColor Yellow
 $DropResult = Invoke-Command -Session $Session1 -ScriptBlock {
 mongosh mongodb://localhost:27017 --quiet --eval 'db.getSiblingDB("TestDB1").dropDatabase()'
 }
 Write-Host "Database dropped: $DropResult" -ForegroundColor Green
} else {
 Write-Host "Database deletion cancelled." -ForegroundColor Yellow
}

Write-Host "`n=== VERIFYING DELETED DATABASE ===" -ForegroundColor Yellow
$ShowResult = Invoke-Command -Session $Session1 -ScriptBlock {
 mongosh mongodb://localhost:27017 --quiet --eval 'db.getSiblingDB("TestDB1")'
 mongosh mongodb://localhost:27017 --quiet --eval 'db.getSiblingDB("TestDB1").getCollectionNames()'
}
Write-Host "Deleted database collections: $ShowResult" -ForegroundColor Green

Output:

=== DELETING DATABASE TO DEMONSTRATE RESTORE PROCESS ===
Type 'DELETE' to drop the test database: DELETE
Dropping test database 'TestDB1' on MongoDB server 1...
Database dropped: { ok: 1, dropped: 'TestDB1' }

=== VERIFYING DELETED DATABASE ===
Deleted database collections: TestDB1 []

The database exists but contains no collections, effectively empty. All our test data is gone. Now we’ll restore it from our snapshots.

Step 5: The Restore Process

Now for the interesting part: restoring from our snapshots. This is a multi-step process:

Write-Host "`n=== RESTORE PROCESS ===" -ForegroundColor Red
Write-Host "WARNING: This will overwrite current data!" -ForegroundColor Red
$Confirm = Read-Host "Type 'YES' to continue with restore"

if ($Confirm -eq 'YES') {
    
    # Stop MongoDB on both servers
 Write-Host "Stopping MongoDB on both servers..." -ForegroundColor Yellow
 Invoke-Command -Session $Session1 -ScriptBlock {
 sudo systemctl stop mongos
 sudo systemctl stop mongod-shard1
 sudo systemctl stop mongod-config
 }
 Invoke-Command -Session $Session2 -ScriptBlock {
 sudo systemctl stop mongod-shard2
 }
    
    # Wait for services to stop
 Start-Sleep -Seconds 5
    
    # Unmount /data on both servers
 Write-Host "Unmounting /data on both servers..." -ForegroundColor Yellow
 Invoke-Command -Session $Session1 -ScriptBlock { sudo umount $using:DataMountPoint }
 Invoke-Command -Session $Session2 -ScriptBlock { sudo umount $using:DataMountPoint }
    
    # Wait for unmount to complete
 Start-Sleep -Seconds 2
    
    # Restore volumes from snapshots
 Write-Host "Restoring volume $FlashArray1DbVol on $ArrayName1..." -ForegroundColor Cyan
 New-Pfa2Volume -Array $FlashArray1 -Name $FlashArray1DbVol `
 -SourceName ($SnapshotFlashArray1.Name + ".$FlashArray1DbVol") -Overwrite $true
    
 Write-Host "Restoring volume $FlashArray2DbVol on $ArrayName2..." -ForegroundColor Cyan
 New-Pfa2Volume -Array $FlashArray2 -Name $FlashArray2DbVol `
 -SourceName ($SnapshotFlashArray2.Name + ".$FlashArray2DbVol") -Overwrite $true
    
    # Wait for volume restore to complete
 Start-Sleep -Seconds 3
    
    # Remount /data on both servers
 Write-Host "Remounting /data on both servers..." -ForegroundColor Yellow
 Invoke-Command -Session $Session1 -ScriptBlock { sudo mount -a }
 Invoke-Command -Session $Session2 -ScriptBlock { sudo mount -a }
    
    # Wait for mount to complete
 Start-Sleep -Seconds 2
    
    # Start MongoDB on both servers
 Write-Host "Starting MongoDB on both servers..." -ForegroundColor Yellow
 Invoke-Command -Session $Session1 -ScriptBlock {
 sudo systemctl start mongod-config
 sleep 3
 sudo systemctl start mongod-shard1
 sleep 3
 sudo systemctl start mongos
 }
 Invoke-Command -Session $Session2 -ScriptBlock {
 sudo systemctl start mongod-shard2
 }
    
    # Wait for services to start
 Start-Sleep -Seconds 10
    
 Write-Host "Restore complete!" -ForegroundColor Green
    
} else {
 Write-Host "Restore cancelled." -ForegroundColor Yellow
}

Output:

=== RESTORE PROCESS ===
WARNING: This will overwrite current data!
Type 'YES' to continue with restore: YES
Stopping MongoDB on both servers...
Unmounting /data on both servers...

Restoring volume vvol-aen-mongo-01-2a764477-vg/Data-8a047da9 on flasharray1.puretec.purestorage.com...

Id                      : cbefd61d-69b9-4b0d-5297-9d0c1dc6fabc
Name                    : vvol-aen-mongo-01-2a764477-vg/Data-8a047da9
ConnectionCount         : 1
Created                 : 11/23/2025 4:20:18 PM
...output omitted...
Source                  : @{Id='cbefd61d-69b9-4b0d-5297-9d0c1dc6fabc'; Name='vvol-aen-mongo-01-2a764477-vg/Data-8a047da9'}
Subtype                 : regular
VolumeGroup             : @{Id='3438324c-909f-df25-d7b5-b2614cfc45d7'; Name='vvol-aen-mongo-01-2a764477-vg'}
Workload                : 

Restoring volume vvol-aen-mongo-02-145f0b6e-vg/Data-a04d05fb on flasharray2.puretec.purestorage.com...

Id                      : 3689fe7e-8555-913e-fd07-6e9af548544a
Name                    : vvol-aen-mongo-02-145f0b6e-vg/Data-a04d05fb
ConnectionCount         : 1
Created                 : 11/23/2025 4:20:18 PM
...output omitted...
Source                  : @{Id='3689fe7e-8555-913e-fd07-6e9af548544a'; Name='vvol-aen-mongo-02-145f0b6e-vg/Data-a04d05fb'}
Subtype                 : regular
VolumeGroup             : @{Id='6ec762d1-4f94-ecea-ce76-f245d9b5d837'; Name='vvol-aen-mongo-02-145f0b6e-vg'}
Workload                : 

Remounting /data on both servers...
Starting MongoDB on both servers...
Restore complete!

Let’s break down what happened during the restore:

5.1: Stop MongoDB Services

We must stop MongoDB in the correct order to avoid corruption:

Order Matters: We stop mongos (the router) first, then shards, then config. This ensures no new operations are routed while we’re shutting down data nodes.

5.2: Unmount Volumes

Unmounting the filesystem disconnects it from the underlying storage volumes, allowing us to restore them safely. This ensures that no file handles are open and that the filesystem cache is flushed.

5.3: Restore Volumes from Snapshots

Next, use the -Overwrite $true parameter to restore the snapshot over the existing volume(s) on each FlashArray. Because Pure Storage FlashArray uses metadata-based snapshots, the restore operation is nearly instantaneous; we’re updating metadata pointers, not copying gigabytes of data.

5.4: Remount and Restart MongoDB

# Remount volumes
Invoke-Command -Session $Session1 -ScriptBlock { sudo mount -a }
Invoke-Command -Session $Session2 -ScriptBlock { sudo mount -a }

# Start MongoDB in correct order
Invoke-Command -Session $Session1 -ScriptBlock {
 sudo systemctl start mongod-config
 sleep 3
 sudo systemctl start mongod-shard1
 sleep 3
 sudo systemctl start mongos
}
Invoke-Command -Session $Session2 -ScriptBlock {
 sudo systemctl start mongod-shard2
}

Startup Sequence: We reverse the shutdown order:

  1. Config server first (the cluster’s metadata store)
  2. Wait 3 seconds for initialization
  3. Shard 1 (depends on config being available)
  4. Wait 3 seconds
  5. Mongos router (needs config and ideally at least one shard)
  6. Shard 2 (can start independently)

These delays ensure each component fully initializes before dependent services start. In production environments, you’d want more sophisticated health checks rather than fixed sleep intervals.

Production Note: Replace fixed sleep intervals with health checks. For example, verify that the config server is responding before starting shards, and verify that at least one shard is healthy before starting mongos.

Step 6: Verification

The moment of truth: Did our restore work?

Write-Host "`n=== VERIFYING RESTORED DATABASE ===" -ForegroundColor Yellow
$ShowResult = Invoke-Command -Session $Session1 -ScriptBlock {
 mongosh mongodb://localhost:27017 --quiet --eval 'db.getSiblingDB("TestDB1")'
 mongosh mongodb://localhost:27017 --quiet --eval 'db.getSiblingDB("TestDB1").getCollectionNames()'
}
Write-Host "Restored database collections: $ShowResult" -ForegroundColor Green

Output:

=== VERIFYING RESTORED DATABASE ===
Restored database collections: TestDB1 [ 'orders', 'products', 'customers' ]

Success! All three collections are back, exactly as they were when we took the snapshot. The distributed data across both FlashArrays has been restored to a consistent point in time.

Key Takeaways

Why This Approach Works

  1. App Consistency: fsyncLock() ensures all data is on disk before snapshots
  2. Atomic Snapshots: Protection groups snapshot all volumes simultaneously
  3. Distributed Coordination: Lock operation propagates across all shards via mongos
  4. Speed: 5-second snapshot window vs. hours for traditional backups
  5. Space Efficiency: Pure Storage’s data reduction means snapshots consume minimal space
  6. Instant Recovery: Metadata-based restore, not data copy operations

Performance Impact

fsyncLock() blocks writes but not reads. Our test showed:

  • Total lock duration: 5.09 seconds
  • Read impact: None
  • Write impact: Blocked for 5.09 seconds

For write-heavy workloads, schedule snapshots during low-activity windows or implement application retry logic.

Conclusion

By coordinating MongoDB’s fsyncLock() with Pure Storage’s protection groups, we achieve:

  • Speed: 5-second snapshot window, near-instant recovery
  • Consistency: App-consistent snapshots across distributed shards and arrays
  • Simplicity: No third-party backup software required

For organizations running MongoDB at scale on Pure Storage, this pattern delivers enterprise-grade data protection with minimal operational overhead.

Resources