How to manage PSA claimrules and SATP rules with esxcli

Shares

I will not try to explain PSA here, there is enough resources to read about it (if you asked me for recommendation I’d suggest Cormac Hogan’s excellent blog) and I don’t feel expert enough to “increase entropy” on that subject.

Instead I’d like to focus on what used to be particularly confusing to me: the claimrules, SATP rules and how to manage them with esxcli.

PSA behaviour can be controlled with two sets of rules (at least I know two of them – maybe there is more 😉 ) first there is set of “claimrules” that decide which PSA plugin (NMP or 3rd party) will handle any particular storage device. Once that is decided there is another set of rules – “SATP rules” which decide which SATP exactly (out of the set of SATPs available in selected MP plugin) will handle the I/O.

Initially I was confusing these two sets because esxcli syntax to manipulate them is very similar and I was easily loosing track of what is going on with PSA on my vSphere host.

Just to set the focus – all listings that will follow come from vSphere 5.5 host in my homelab.

NOTE: do not forget to do vmware backup before you start playing with PSA rules 🙂

So let’s have a look at “claimrules” first.

~ # esxcli storage core plugin list
Plugin name  Plugin class
-----------  ------------
NMP          MP
~ #
~ # esxcli storage core claimrule list
Rule Class   Rule  Class    Type       Plugin     Matches
----------  -----  -------  ---------  ---------  ---------------------------------
MP              0  runtime  transport  NMP        transport=usb
MP              1  runtime  transport  NMP        transport=sata
MP              2  runtime  transport  NMP        transport=ide
MP              3  runtime  transport  NMP        transport=block
MP              4  runtime  transport  NMP        transport=unknown
MP            101  runtime  vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP            101  file     vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP          65535  runtime  vendor     NMP        vendor=* model=*
~ #
~ # esxcli storage core path list --path=vmhba33:C0:T0:L2
iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434
   UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434
   Runtime Name: vmhba33:C0:T0:L2
   Device: t10.F405E46494C45425930586A5B426D2133533D6D276559434
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C45425930586A5B426D2133533D6D276559434)
   Adapter: vmhba33
   Channel: 0
   Target: 0
   LUN: 2
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:host03-208b6b7a
   Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:host03-208b6b7a
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:mylabfiler01.tier1 Alias= Session=00023d000001 PortalTag=1

As you can see I only have Native Multipathing Plugin (NMP) installed, the claimrule list is also pretty generic, with either NMP or MASK_PATH plugins applied and MASK_PATH is used to hide some specific (unsupported?) SAN device from vSphere. These claimrules are checked in descending order (so if you configure more than one rule that could match a device the one that is listed first will apply).
You can also see a “catch all” rule at the bottom of the list, it is there to make sure that any device detected yet not matched with previous rules is handled with NMP.

I also show information about specific storage path (which happens to be iSCSI LUN that I share from OpenFiler appliance) because I want to show how to change claimrules to mask this particular LUN.

~ # esxcli storage core claimrule add --autoassign --type=target --transport=iscsi --iqn iqn.2006-01.com.openfiler:mylabfiler01.tier1 --lun 2 --plugin=MASK_PATH
~ # esxcli storage core claimrule list
Rule Class   Rule  Class    Type       Plugin     Matches
----------  -----  -------  ---------  ---------  ------------------------------------------------------------------
MP              0  runtime  transport  NMP        transport=usb
MP              1  runtime  transport  NMP        transport=sata
MP              2  runtime  transport  NMP        transport=ide
MP              3  runtime  transport  NMP        transport=block
MP              4  runtime  transport  NMP        transport=unknown
MP            101  runtime  vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP            101  file     vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP           5001  file     target     MASK_PATH  transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2
MP          65535  runtime  vendor     NMP        vendor=* model=*
~ # 
~ # esxcli storage core claimrule load
~ # esxcli storage core claimrule list
Rule Class   Rule  Class    Type       Plugin     Matches
----------  -----  -------  ---------  ---------  ------------------------------------------------------------------
MP              0  runtime  transport  NMP        transport=usb
MP              1  runtime  transport  NMP        transport=sata
MP              2  runtime  transport  NMP        transport=ide
MP              3  runtime  transport  NMP        transport=block
MP              4  runtime  transport  NMP        transport=unknown
MP            101  runtime  vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP            101  file     vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP           5001  runtime  target     MASK_PATH  transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2
MP           5001  file     target     MASK_PATH  transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2
MP          65535  runtime  vendor     NMP        vendor=* model=*
~ #
~ # esxcli storage core claiming reclaim --device=t10.F405E46494C45425930586A5B426D2133533D6D276559434
~ #
~ # ~ # esxcli storage core path list --path=vmhba33:C0:T0:L2
iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-
   UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-
   Runtime Name: vmhba33:C0:T0:L2
   Device: No associated device
   Device Display Name: No associated device
   Adapter: vmhba33
   Channel: 0
   Target: 0
   LUN: 2
   Plugin: MASK_PATH
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:host03-208b6b7a
   Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
   Maximum IO Size: 131072
~ #

I use esxcli storage core claimrule add command here to create new claimrule, let vSphere autoassign the rule number, then I specify type of “target” and provide as much details as possible that will allow device to be matched (so I declare the protocol needs to be iSCSI, coming from specific iqn with LUN 2)
I want this LUN to be masked, so I select MASK_PATH plugin.
The rule shows immediately in the list, yet only in “file” class, which means it is saved, but vmkernel is not using it at the moment and we need to (re)load the claimrules.
The final step after loading claimrules (our claimrule is listed twice now, both in “file” and “runtime” class) is to reclaim the device (to activate our new rule) and… we’re ready.
Information about the path shows now that plugin has changed to MASK_PATH and the path state is DEAD!

You have to believe me that this device disappears from vSphere Client immediately after what we just did.

OK, let’s try to revert this now and make LUN 2 visible again.

~ # esxcli storage core claimrule remove --rule=5001
~ # esxcli storage core claimrule list
Rule Class   Rule  Class    Type       Plugin     Matches
----------  -----  -------  ---------  ---------  ------------------------------------------------------------------
MP              0  runtime  transport  NMP        transport=usb
MP              1  runtime  transport  NMP        transport=sata
MP              2  runtime  transport  NMP        transport=ide
MP              3  runtime  transport  NMP        transport=block
MP              4  runtime  transport  NMP        transport=unknown
MP            101  runtime  vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP            101  file     vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP           5001  runtime  target     MASK_PATH  transport=iscsi iqn=iqn.2006-01.com.openfiler:mylabfiler01.tier1 lun=2
MP          65535  runtime  vendor     NMP        vendor=* model=*
~ #
~ # esxcli storage core claimrule load
~ # esxcli storage core claimrule list
Rule Class   Rule  Class    Type       Plugin     Matches
----------  -----  -------  ---------  ---------  ---------------------------------
MP              0  runtime  transport  NMP        transport=usb
MP              1  runtime  transport  NMP        transport=sata
MP              2  runtime  transport  NMP        transport=ide
MP              3  runtime  transport  NMP        transport=block
MP              4  runtime  transport  NMP        transport=unknown
MP            101  runtime  vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP            101  file     vendor     MASK_PATH  vendor=DELL model=Universal Xport
MP          65535  runtime  vendor     NMP        vendor=* model=*
~ #
~ # esxcli storage core claiming reclaim --device=t10.F405E46494C45425930586A5B426D2133533D6D276559434
Unknown Device name t10.F405E46494C45425930586A5B426D2133533D6D276559434
~ #
~ # Duh! That's confusing, let's try other method (using paths)
~ #
~ # esxcli storage core claiming unclaim --type=path --path=vmhba33:C0:T0:L2
~ # esxcli storage core path list --path=vmhba33:C0:T0:L2
iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-
   UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-
   Runtime Name: vmhba33:C0:T0:L2
   Device: No associated device
   Device Display Name: No associated device
   Adapter: vmhba33
   Channel: 0
   Target: 0
   LUN: 2
   Plugin: (unclaimed)
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:host03-208b6b7a
   Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
   Maximum IO Size: 131072
~ #
~ # esxcli storage core adapter rescan --all
~ # esxcli storage core path list --path=vmhba33:C0:T0:L2
iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434
   UID: iqn.1998-01.com.vmware:host03-208b6b7a-00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1-t10.F405E46494C45425930586A5B426D2133533D6D276559434
   Runtime Name: vmhba33:C0:T0:L2
   Device: t10.F405E46494C45425930586A5B426D2133533D6D276559434
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C45425930586A5B426D2133533D6D276559434)
   Adapter: vmhba33
   Channel: 0
   Target: 0
   LUN: 2
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:deobhv03-208b6b7a
   Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:mylabfiler01.tier1,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:host03-208b6b7a
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:mylabfiler01.tier1 Alias= Session=00023d000001 PortalTag=1
   Maximum IO Size: 131072
~ #

Now “unmasking” a LUN that we’ve hidden with MASK_PATH plugin can be a bit tricky, at the beginning we need to remove the rule (using just the rule number) from wherever it is stored, but as you can see “remove” operation does not immediately deactivate the claimrule (it is still listed with “runtime” class).
This is solved by (re)loading the whole claimrule set, but subsequent attempt to reclaim the device fails with “Unknown Device” error…
Well it makes sense, we’ve just masked it a moment ago, so t10 identifier we use disappeared from our system…

At first I thought host reboot is the only solution to make that LUN visible again, but after a few minutes of playing around with esxcli I discovered there is a way to avoid a reboot.

We can not reclaim a device, but we can set the storage path (using runtime identifier) that “leads” to device to be “unclaimed” and then rescan our storage adapters so that default “catch all” claimrule is matched and our LUN gets handled by NMP again, you can see all the steps listed above.

The thing worth to remember is that claimrules are really easy – it is just about which MP plugin to assign to storage devices.

SATP rules on the other hand can get really complicated, cause you can have many SATPs as “sub-plugins” to MP plugin that is selected. NMP itself (so the PSA plugin that you get “out-of-box”, just with “vanilla” vSphere installation) has 13 of these SATPs, as a result SATP rule set gets much bigger and if you consider that each SAN can require some specific options to be provided by the means of SATP rules… Well I suggest you just to execute

 esxcli storage nmp satp rule list

from your host’s console and admire this whole list scrolling.

Yet the esxcli syntax to manipulate these SATP rules is very similar to what we exercised with claimrules (and this is what was confusing me a lot when I first started to use them).

As an example I will show you how to “make” an SSD out of traditional, magnetic hard disk (something you can probably hear about at every VMware training, also “tier-1” bloggers like Duncan or William have posted about it like ages ago).

# esxcli storage nmp satp rule add --satp=VMW_SATP_LOCAL --device=mpx.vmhba1:C0:T1:L0 --option=enable_ssd
# esxcli storage nmp satp rule list --satp=VMW_SATP_LOCAL
Name            Device               Vendor  Model  Driver  Transport  Options     Rule Group  Claim Options  Default PSP  PSP Options  Description
--------------  -------------------  ------  -----  ------  ---------  ----------  ----------  -------------  -----------  -----------  ----------------------------
VMW_SATP_LOCAL                                              usb                    system                                               USB Devices
VMW_SATP_LOCAL                                              ide                    system                                               IDE Devices
VMW_SATP_LOCAL                                              block                  system                                               RAID Block Devices
VMW_SATP_LOCAL                                              parallel               system                                               Parallel SCSI Devices
VMW_SATP_LOCAL                                              sas                    system                                               Serial Attached SCSI Devices
VMW_SATP_LOCAL                                              sata                   system                                               Serial ATA Devices
VMW_SATP_LOCAL                                              unknown                system                                               Unknown Devices
VMW_SATP_LOCAL  mpx.vmhba1:C0:T1:L0                                    enable_ssd  user
#
# esxcli storage core claiming reclaim --device=mpx.vmhba1:C0:T1:L0
# esxcli storage core device list --device=mpx.vmhba1:C0:T1:L0
mpx.vmhba1:C0:T1:L0
   Display Name: Local VMware, Disk (mpx.vmhba1:C0:T1:L0)
   Has Settable Display Name: false
   Size: 40960
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0
   Vendor: VMware,
   Model: VMware Virtual S
   Revision: 1.0
   SCSI Level: 2
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unsupported
   Other UIDs: vml.0000000000766d686261313a313a30
   Is Local SAS Device: false
   Is Boot USB Device: false
   No of outstanding IOs with competing worlds: 32
#

The only thing we do here is to add a rule that assigns VMW_SATP_LOCAL and sets “enable_ssd” option for our mpx.vmhba1:C0:T1:L0 device. Luckily we don’t need to (re)load anything just reclaim the device – and here we go, our magnetic drive will be considered SSD by vSphere from now on (you can see “Is SSD” property has value of “true”).

To be honest – reclaiming device just for SATP rule change also used to confuse me a lot – even though we only modify some SATP option it looks like whole PSA “stack” needs to be reset for this operation (including initial MP plugin selection for device we are working on).

The other observation I have – it seems like SATP rules are applied in different manner than “claimrules”, I’ve written above that “claimrules” are applied in descending order (and they have numbers!), as for SATP I think these rules are applied according to “best match” policy (same way like network gateway to forward traffic to is selected if you have multiple TCP/IP static routes defined). If you look at SATP rules list for VMW_SATP_LOCAL above there are at least two rules (“transport = sata” configured by default and our “device = mpx.vmhba1:C0:T1:L0” rule) that match, yet our trick works, so it seems “best match” policy is used (cause our rule is more precise, so makes better match than generic “transport” class rule).
If not “best match” policy, maybe the order in which rule classes are presented in listing above (Device > Vendor > Model > Driver > Transport) is the key: so if you include device name in your rule definition it overrides all rules with matching vendor or driver?
Oh and have you noticed? SATP rules don’t have numbers!

But again – this is only an “educated guess” based on some of my experiments, I wasn’t able to find any documentation that would prove me right or wrong and explain in what order SATP rules are really applied.

OK, it is fun to make a fool of vmkernel for a while, but just having SSD enabled for a spindle hard drive will not give you any performance boost, in fact it will give you nothing but trouble if you continue like this.
The only real-life scenario you might want to use this trick is when you have SSD from some “out-of-mainstream” vendor and capabilities of this disk are not recognized properly by vSphere, then you have to configure this SATP rule to get the performance you paid for.

Since I used this rule against magnetic disk, I’m going to revert my settings to their original state.

~ # esxcli storage nmp satp rule remove --satp=VMW_SATP_LOCAL --device=mpx.vmhba1:C0:T1:L0 --option=enable_ssd
~ # esxcli storage core claiming reclaim --device=mpx.vmhba1:C0:T1:L0
~ # esxcli storage core device list --device=mpx.vmhba1:C0:T1:L0
mpx.vmhba1:C0:T1:L0
   Display Name: Local VMware, Disk (mpx.vmhba1:C0:T1:L0)
   Has Settable Display Name: false
   Size: 40960
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/mpx.vmhba1:C0:T1:L0
   Vendor: VMware,
   Model: VMware Virtual S
   Revision: 1.0
   SCSI Level: 2
   Is Pseudo: false
   Status: on
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unsupported
   Other UIDs: vml.0000000000766d686261313a313a30
   Is Local SAS Device: false
   Is Boot USB Device: false
   No of outstanding IOs with competing worlds: 32

Removing SATP rules is not as easy as with claimrules – you can not use rule number (they don’t have numbers, right?), so there is a need to provide the exact syntax the rule was created with.
Once the rule is removed – just reclaim the device again and everything is back to normal.

I hope you find this lengthy post useful, any comments are welcome!

0 0 votes
Article Rating

Sebastian Baryło

Successfully jumping to conclusions since 2001.

Subscribe
Notify of
guest
4 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Guest

Small digression. Would you be able to configure PSA related settings via host profile? It might be useful if you think about VCAP examination. Really.

Arel

I have learnt to vcp exam and this post excellent explain claimrules to me 🙂 thanks 🙂

4
0
Would love your thoughts, please comment.x
()
x

FOR FREE. Download Nutanix port diagrams

Join our mailing list to receive an email with instructions on how to download 19 port diagrams in MS Visio format.

NOTE: if you do not get an email within 1h, check your SPAM filters

You have Successfully Subscribed!

Don't miss any posts. Subscribe To Our Newsletter!!

Join our mailing list to receive the latest news and updates from www.vmwaremine.com.

You have Successfully Subscribed!

Pin It on Pinterest