Using the Pure Storage PowerShellSDK2 - Part 3 - Getting Performance Data from FlashArray

Page content

Welcome back to the third installment of our blog series on using the Pure Storage PowerShell SDK2. In this post, we’ll learn how to retrieve performance data from FlashArray and Cloud Block Store. Here, you’ll uncover the intricacies of extracting performance data across several object types, including Volumes and Hosts. We will dig into the object model that exposes crucial performance insights. Moreover, we’ll delve into the realm of performance analysis, addressing common customer questions such as:

  • Identifying the top 10 volumes in terms of IOPS or latency.
  • Pinpointing the top 10 hosts in terms of IOPS or latency.
  • Finding the top 10 volumes in terms of IOPs during specific time intervals in the past.

While Pure1, the Purity web interface, and our Open Metrics Exporter for FlashArray our platforms are capable of providing this information effortlessly in several ways. This post will empower you with the skills to retrieve and extract this performance data using the Pure Storage PowerShell SDK2. With this, you can seamlessly perform ad hoc analysis or integrate this data into other analytical tools to better understand your storage performance.

Finding the Performance cmdlets

We will start with discovering which cmdlets expose performance data to us and for which objects in the array. If we look at the cmdlets with the string performance in the name, we can see what’s available to us to work with. In the listing below, you can see several object types, Array, Directory, Host Group, Host, NetworkInterface, Pod, Volume Group, and Volume.

This post focuses on Volumes and Hosts. We can answer those common customer questions we mentioned in the previous section. We can also use the techniques in this post to analyze performance against these other object types since the performance information exposed is nearly the same for all objects.

Get-Command -Module PureStoragePowerShellSDK2 | Where-Object { $_.Name -like "*performance" } 

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Cmdlet          Get-Pfa2ArrayPerformance                           2.24.70    PureStoragePowerShellSDK2
Cmdlet          Get-Pfa2DirectoryPerformance                       2.24.70    PureStoragePowerShellSDK2
Cmdlet          Get-Pfa2HostGroupPerformance                       2.24.70    PureStoragePowerShellSDK2
Cmdlet          Get-Pfa2HostPerformance                            2.24.70    PureStoragePowerShellSDK2
Cmdlet          Get-Pfa2NetworkInterfacePerformance                2.24.70    PureStoragePowerShellSDK2
Cmdlet          Get-Pfa2PodPerformance                             2.24.70    PureStoragePowerShellSDK2
Cmdlet          Get-Pfa2VolumeGroupPerformance                     2.24.70    PureStoragePowerShellSDK2
Cmdlet          Get-Pfa2VolumePerformance                          2.24.70    PureStoragePowerShellSDK2

Understanding the Resource Performance Object Model

First, I want to show you what performance information you can retrieve from FlashArray or Cloud Block Store via the PowerShell SDK2 module. In the last section, I showed you some cmdlets that exposed performance information for the many objects available in FlashArray and Cloud Block Store. In this section, let’s look at the cmdlet that exposes Volume information with Get-Pfa2VolumePerformance. In the output below, you’ll find that the type is PureStorage.FlashArray.Rest.Model.ResourcePerformance, and you can see performance information across many dimensions exposed as Properties.

Get-Pfa2VolumePerformance -Array $FlashArray | Get-Member

   TypeName: PureStorage.FlashArray.Rest.Model.ResourcePerformance

Name                               MemberType Definition
----                               ---------- ----------
BaseToString                       Method     string BaseToString(string[] names, System.Object[] values)
CompareMembers                     Method     bool CompareMembers(System.Object lhs, System.Object rhs)
Equals                             Method     bool Equals(System.Object input), bool Equals(PureStorage.FlashArray.Rest.Model.ResourcePerformance input), bool IEquatable[ResourcePerformance].Equals(PureStorage.FlashArray.Rest.Model.ResourcePerformance other)
GetHashCode                        Method     int GetHashCode()
GetType                            Method     type GetType()
GetValue                           Method     long GetValue(System.Object value)
HashMembers                        Method     int HashMembers(int hashCode, int value), int HashMembers(int hashCode, long value), int HashMembers[T](int hashCode, T value)
ToString                           Method     string ToString()
BytesPerMirroredWrite              Property   System.Nullable[long] BytesPerMirroredWrite {get;set;}
BytesPerOp                         Property   System.Nullable[long] BytesPerOp {get;set;}
BytesPerRead                       Property   System.Nullable[long] BytesPerRead {get;set;}
BytesPerWrite                      Property   System.Nullable[long] BytesPerWrite {get;set;}
Id                                 Property   string Id {get;}
MirroredWriteBytesPerSec           Property   System.Nullable[long] MirroredWriteBytesPerSec {get;set;}
MirroredWritesPerSec               Property   System.Nullable[long] MirroredWritesPerSec {get;set;}
Name                               Property   string Name {get;set;}
QosRateLimitUsecPerMirroredWriteOp Property   System.Nullable[long] QosRateLimitUsecPerMirroredWriteOp {get;set;}
QosRateLimitUsecPerReadOp          Property   System.Nullable[long] QosRateLimitUsecPerReadOp {get;set;}
QosRateLimitUsecPerWriteOp         Property   System.Nullable[long] QosRateLimitUsecPerWriteOp {get;set;}
QueueUsecPerMirroredWriteOp        Property   System.Nullable[long] QueueUsecPerMirroredWriteOp {get;set;}
QueueUsecPerReadOp                 Property   System.Nullable[long] QueueUsecPerReadOp {get;set;}
QueueUsecPerWriteOp                Property   System.Nullable[long] QueueUsecPerWriteOp {get;set;}
ReadBytesPerSec                    Property   System.Nullable[long] ReadBytesPerSec {get;set;}
ReadsPerSec                        Property   System.Nullable[long] ReadsPerSec {get;set;}
SanUsecPerMirroredWriteOp          Property   System.Nullable[long] SanUsecPerMirroredWriteOp {get;set;}
SanUsecPerReadOp                   Property   System.Nullable[long] SanUsecPerReadOp {get;set;}
SanUsecPerWriteOp                  Property   System.Nullable[long] SanUsecPerWriteOp {get;set;}
ServiceUsecPerMirroredWriteOp      Property   System.Nullable[long] ServiceUsecPerMirroredWriteOp {get;set;}
ServiceUsecPerReadOp               Property   System.Nullable[long] ServiceUsecPerReadOp {get;set;}
ServiceUsecPerReadOpCacheReduction Property   System.Nullable[float] ServiceUsecPerReadOpCacheReduction {get;set;}
ServiceUsecPerWriteOp              Property   System.Nullable[long] ServiceUsecPerWriteOp {get;set;}
Time                               Property   System.Nullable[datetime] Time {get;set;}
UsecPerMirroredWriteOp             Property   System.Nullable[long] UsecPerMirroredWriteOp {get;set;}
UsecPerReadOp                      Property   System.Nullable[long] UsecPerReadOp {get;set;}
UsecPerWriteOp                     Property   System.Nullable[long] UsecPerWriteOp {get;set;}
WriteBytesPerSec                   Property   System.Nullable[long] WriteBytesPerSec {get;set;}
WritesPerSec                       Property   System.Nullable[long] WritesPerSec {get;set;}

Tip: Examine the other object types’ (Array, Hosts, Host Groups, and more) and their properties using the performance cmdlets we discovered previous section and piping that into Get-Member.

Deep Dive on the Performance Metrics

Now, there are a lot of performance metrics exposed here. Below is a table summarizing the performance metrics interesting to this blog post with a description. There are additional metrics for advanced scenarios like ActiveCluster and user-defined QoS policies.

There are three columns in this table. The PowerShell property is the actual property name the PowerShell object returns. The API Property is the actual REST API property. There is also a short description of what the performance metrics mean. The PowerShell object property is camelcase, and the array API response property has a different format, separating parts of the name with underscores _. For example, ReadsPerSec is the PowerShell property, and reads_per_sec is the array API response property. Notice the underscores are removed from the PowerShell property, and the API response property is lowercase and is case sensitive.

As discussed in our previous post Using the Pure Storage PowerShellSDK2 - Part 2 - Working With Data when working with data on the client side you will use the PowerShell Property. When working with data inside the REST API, you will use the API Property.

TIP: If you’re having trouble finding the right API property, visit the FlashArray REST API Reference Guides.

PowerShell Property API Property Description
BytesPerOp bytes_per_op Average I/O size measured in bytes, both read and write
BytesPerRead bytes_per_read Average read I/O size measured in bytes
BytesPerWrite bytes_per_write Average write I/O size measured in bytes
Id id A unique identifier for the object
Name name The user friendly name of the object
QueueUsecPerReadOp queue_usec_per_read_op Average time that a read I/O request spends in the array waiting to be served, measured in microseconds
QueueUsecPerWriteOp queue_usec_per_write_op Average time that a write IO request spends in the array waiting to be served, measured in microseconds
ReadBytesPerSec read_bytes_per_sec The number of bytes read per second
ReadsPerSec reads_per_sec Number of read requests processed per second, read IOPs
SanUsecPerReadOp san_usec_per_read_op Time required transferring data between initiator and FlashArray for read IOs, measured in microseconds
SanUsecPerWriteOp san_usec_per_write_op Time required transferring data between initiator and FlashArray for write IOs, measured in microseconds
ServiceUsecPerReadOp service_usec_per_read_op The average time required for the array to service a read request. Measured in microseconds.
ServiceUsecPerWriteOp service_usec_per_write_op The average time required for the array to service a write request. Measured in microseconds.
Time time The time when the sample performance data was taken. Measured in milliseconds since the UNIX epoch.
UsecPerReadOp usec_per_read_op The average time it takes the array to process an I/O read request, measured in microseconds. The average time does not include SAN time, queue time, or QoS rate limit time.
UsecPerWriteOp usec_per_write_op The average time it takes the array to process an I/O write request, measured in microseconds. The average time does not include SAN time, queue time, or QoS rate limit time.
WriteBytesPerSec write_bytes_per_sec Amount of data written per second
WritesPerSec writes_per_sec Number of write requests processed per second

Table 1: Performance Metrics Exposed via PowerShell

Tip: When working with latency metrics, latency is broken out into components measuring different latency areas in the I/O path, such as SAN Time, Service Time, Queue Time, and QoS Time. Descriptions are in the above chart. This is immensely useful for troubleshooting as you can isolate where you’re latency is occurring.

Kick Off a Workload

I will kick off a workload so we can have meaningful data to look at and analyze. Below is the code to kick off a SQL Server database backup using the dbatools PowerShell module. In your environment, you likely have workloads running. In my lab…I need to provide something interesting for us to look at here ;)

Start-Job -ScriptBlock {
    $username = "sa"
    $password = 'S0methingS@Str0ng!' | ConvertTo-SecureString -AsPlainText
    $SqlCredential = new-object -typename System.Management.Automation.PSCredential -argumentlist $username, $password
    Backup-DbaDatabase -SqlInstance 'aen-sql-22-a' -SqlCredential $SqlCredential -Database 'FT_Demo' -Type Full -CopyOnly -FilePath NUL 

Finding Busy Volumes

Using our API sorting method discussed in our previous post Using the Pure Storage PowerShellSDK2 - Part 2 - Working With Data, in the code below I’m using Get-Pfa2VolumePerformance and the -Sort parameter to sort by reads_per_sec- descending. And then, I’m using the -Limit parameter to return only the top 10 volumes.

I’m then piping that output to Select-Object Name, Time, ReadsPerSec, BytesPerRead so that I’m returning only the name, the time of the sample, the read IOPs, and the average read IO size in bytes. Who is the top culprit in terms of read IO, that volume supporting the SQL Server database backup I kicked off earlier in the post, vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242. This volume is reading 3586 read IOs/sec with an average read IO size of around 512KB.

Get-Pfa2VolumePerformance -Array $FlashArray -Sort 'reads_per_sec-' -Limit 10 | 
    Select-Object Name, Time, ReadsPerSec, BytesPerRead

Name                                          Time                 ReadsPerSec BytesPerRead
----                                          ----                 ----------- ------------
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242 10/6/2023 6:30:30 PM        3586       524288
PRODDB003                                     10/6/2023 6:30:30 PM         920       143206
fds23123123                                   10/6/2023 6:30:30 PM          21       421522
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-47094663 10/6/2023 6:30:30 PM          19        23067
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01       10/6/2023 6:30:30 PM           8        80896
vvol-catalog-18098d57-vg/Config-11b28d2b      10/6/2023 6:30:30 PM           5          614
SQLFB-REP                                     10/6/2023 6:30:30 PM           0            0
SQLFB-DB1                                     10/6/2023 6:30:30 PM           0            0
SQLFB-DB2                                     10/6/2023 6:30:30 PM           0            0
SQLFB-DB3                                     10/6/2023 6:30:30 PM           0            0

Note: The default resolution on storage objects like Volumes is a 30-second window starting when the cmdlet is executed.

Using this powerful PowerShell technique, you can quickly sort your way through performance data across any of the performance metrics mentioned that we learned about in the previous section and also across the many different object types in FlashArray or Cloud Block Store such as Host, Host Groups and the others listed in the section.

But Where is Total IOPs?

In the object model we just went through, you see ReadsPerSec and WritesPerSec, but there isn’t a property for total IOPs. In this case, we must calculate that locally on the client side. So, in the code below, I’m taking a performance sample with Get-Pfa2VolumePerformance and storing that in the variable $VolumePerformance. From there, we can use the data in that variable in local memory to perform our calculations quickly. And that’s what we have below. I’m adding ReadsPerSec to WritesPerSec and creating a new calculated property, IOsPerSec. Once the calculation completes, that’s passed across the pipeline to Sort-Object -Property IOsPerSec -Descending to sort by the IOsPerSec, and then we use Select-Object -First 10 to limit the output to the top 10.

$VolumePerformance = Get-Pfa2VolumePerformance -Array $FlashArray
$VolumePerformance | 
    Select-Object Name, ReadsPerSec, WritesPerSec, @{label="IOsPerSec";expression={$_.ReadsPerSec + $_.WritesPerSec}} | 
    Sort-Object -Property IOsPerSec -Descending | 
    Select-Object -First 10

Name                                          ReadsPerSec WritesPerSec IOsPerSec
----                                          ----------- ------------ ---------
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242        3199            0      3199
vvol-MUTTDB001-bd7156db-vg/Data-514e94f7                0          732       732
sn1-m70-f06-33-vc01-ds02                               42          115       157
sn1-m70-f06-33-placeholder                             90            9        99
sn1-m70-f06-33-vc01-ds01                               12           59        71
vvol-catalog-18098d57-vg/Config-11b28d2b               15           33        48
pxclouddrive-ec24204b91                                 2           20        22
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01                16            0        16
vvol-MUTTDB001-bd7156db-vg/Data-fa47a22c               11            0        11
vvol-ds3e2-fsa-lab-dc2639f4-vg/Data-c853f680            0            6         6

So, there are some scenarios where you’ll need to do client-side calculations. Here we see an example around IOPS another example is performing calculations on the latency metrics. Bringing the performance data locally and performing the calculations in memory can provide excellent performance since it will minimize the number of round trips to the array. This data is a point-in-time representation of the performance data, you will need to sample the data again to get a fresh sample.

Finding Busy Hosts

So, now that you know the object model and how to work with performance data, let’s look at one more example. Let’s find the top 10 busiest hosts on the array by read IOPs. Most of what we will walk through below resembles the Volume example above. The ResourcePerformance objects are nearly identical between the different object types such as Volume and Host. Let’s jump right into the code and show you how to find the top 10 busiest hosts on your FlashArray or Cloud Block Store in terms of read IOPs.

Using the cmdlet Get-Pfa2HostPerformance, we can use FlashArray’s API to -Sort on our API property reads_per_sec and Limit the output to the top 10…we then pipe the output to Select-Object Name, Time, ReadsPerSec, BytesPerRead to select the exact properties we want and there’s the listing.

Get-Pfa2HostPerformance -Array $FlashArray -Sort 'reads_per_sec-' -Limit 10 | 
Select-Object Name, Time, ReadsPerSec, BytesPerRead

Name                           Time                 ReadsPerSec BytesPerRead
----                           ----                 ----------- ------------
sn1-r720-f06-09                10/6/2023 7:52:02 PM        2634       110026
sn1-r720-f06-13                10/6/2023 7:52:02 PM        1199        65556
sn1-cfi-c08-15-c01-b02         10/6/2023 7:52:02 PM         108        18849
sn1-cfi-c08-15-c01-b04         10/6/2023 7:52:02 PM          96        19456
sn1-cfi-c08-15-c01-b03         10/6/2023 7:52:02 PM          81        13754
sn1-x70-f06-27:sn1-r720-f07-03 10/6/2023 7:52:02 PM           5        78643
sn1-r720-f07-03                10/6/2023 7:52:02 PM           2        65536
node-1-1                       10/6/2023 7:52:02 PM           1        16384
sn1-r720-f06-15                10/6/2023 7:52:02 PM           0            0
sn1-r720-f07-01                10/6/2023 7:52:02 PM           0            0

Let’s Learn How to Look Back in Time…

So now you know how to get performance data for different objects, such as Volume and Hosts, from FlashArray or Cloud Block Store. By default, this data is for a sample from the time of the cmdlet execution and a window of 30 seconds back. So that’s good for answering the questions regarding what’s happening in your array right now, but now let’s talk about how to look back in time so that we can answer the question of what happened when.

All of the performance cmdlets we learned about above expose three parameters to help with this

  • -StartTime - Displays historical performance data for the specified time window, where StartTime is the beginning of the time window.
  • -EndTime - Displays historical performance data for the specified time window, where EndTime is the end of the time window.
  • -Resolution - The number of milliseconds between samples of historical data.

Here, I will show you how to use a fixed time window defining both parameters StartTime and EndTime. There are some additional use cases around how one can use these parameters. Please read the help for the cmdlets to learn more.

In the code below, let’s find the top 10 busiest volumes sorted by read IOPs, looking back at a one-day window that started two days ago. When looking back in time, you must define a Resolution for your sampling. I’m using 1800000 milliseconds, which is 30 minutes.

With the date math on the first three lines, we’re defining variables $StartTime and $EndTime for the parameters -StartTime and -EndTime.

We then use Get-Pfa2VolumePerformance again and add in these three parameters, StartTime, EndTime and Resolution. What’s returned is the top 10 volumes in the array for a one-day window, starting three days ago and ending two days ago. We then pipe that into Select-Object to filter the output to the results we want to output.

$Today = (Get-Date).ToUniversalTime()
$StartTime = $Today.AddDays(-3)
$EndTime = $Today.AddDays(-2)

Get-Pfa2VolumePerformance -Array $FlashArray -Sort 'reads_per_sec-' -Limit 10 -StartTime $StartTime -EndTime $EndTime -resolution 1800000 |
    Select-Object Name, Time, ReadsPerSec

Name                                           Time                 ReadsPerSec
----                                           ----                 -----------
PRODDB003                                      10/3/2023 6:59:22 PM         344
vvol-MUTTDB001-bd7156db-vg/Data-514e94f7       10/3/2023 6:59:22 PM         194
vvol-MUTTDB001-bd7156db-vg/Data-fa47a22c       10/3/2023 6:59:22 PM         160
vc01-mgmt-pod::sn1-x70-m70-vc01-mgmt-01        10/3/2023 6:59:22 PM          60
sn1-m70-f06-33-vc01-ds02                       10/3/2023 6:59:22 PM          53
sn1-m70-f06-33-vc01-ds01                       10/3/2023 6:59:22 PM          34
vvol-catalog-18098d57-vg/Config-11b28d2b       10/3/2023 6:59:22 PM          30
px-fsa-dr-01                                   10/3/2023 6:59:22 PM          28
vvol-MUTTDB001-bd7156db-vg/Data-e7199a87       10/3/2023 6:59:22 PM          27
SAPHANA-ActiveDR2::ryan-saphana-activedr-site2 10/3/2023 6:59:22 PM          16

We can extend this search by combining other techniques we learned in our previous post and add in a -Filter. This allows us to focus our search on a subset of objects. So rather than looking at all volumes in the array, we’re limiting our search to volume names with the string *aen-sql-22-a* in the name, which is all the volumes associated with a specific SQL Server in my lab. This enables us to track performance metrics for specific volumes in our array.

Get-Pfa2VolumePerformance -Array $FlashArray -Limit 10 -StartTime $StartTime -EndTime $EndTime -resolution 1800000 -Filter "name='*aen-sql-22-a*'" -Sort 'reads_per_sec-'  | 
    Sort-Object ReadsPerSec -Descending |
    Select-Object Name, Time, ReadsPerSec

Name                                            Time                 ReadsPerSec
----                                            ----                 -----------
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-47094663   10/3/2023 6:59:22 PM           6
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-47094663   10/3/2023 7:29:22 PM           6
vvol-aen-sql-22-a-1-3d9acfdd-vg/Config-81e9c6b8 10/3/2023 6:59:22 PM           0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-620d202b   10/3/2023 6:59:22 PM           0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-701193f8   10/3/2023 6:59:22 PM           0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-77084035   10/3/2023 6:59:22 PM           0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-87ee3d7c   10/3/2023 6:59:22 PM           0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Data-cabce242   10/3/2023 6:59:22 PM           0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Swap-80dfb009   10/3/2023 6:59:22 PM           0
vvol-aen-sql-22-a-1-3d9acfdd-vg/Config-81e9c6b8 10/3/2023 7:29:22 PM           0

Wrapping Things Up

To wrap up this third installment of my Pure Storage PowerShell SDK2 Blog Series, we’ve learned how to effectively retrieve performance data from FlashArray and Cloud Block Store using the Pure Storage PowerShell SDK2. We’ve gained insights into storage performance by exploring various object types, such as Volumes and Hosts. This knowledge allows you to conduct ad-hoc analysis and integrate the data into other analytical tools to understand your storage performance better. Finally, I’ve demonstrated how to analyze historical performance data using time-based parameters.

Pure Storage PowerShell SDK2 Blog Series

This article is part of a blog series covering the Pure Storage PowerShell SDK2. Check out the other posts in this series:

You can find the supporting code for this blog series at this GitHub Repo.