VMware: SCSI reservation conflicts
Last week I had some trouble by a customer, 4 of my 8 datastores aren’t visible/accessible on the 6 ESX 3.5u2 hosts connected to a (FC) HP MSA1500. Some datastores become unavailable and some were not affected. Numerous VMs were down, some of those with warning messages like Orphaned and Inaccessible.
Oke.. let’s troubleshoot:
Checking for active paths: esxcfg-mpath -l | grep -i active
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:2 On active preferred
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:6 On active preferred
Local 70:0.0 vmhba2:0:0 On active preferred
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:1 On active preferred
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:5 On active preferred
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:3 On active preferred
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:7 On active preferred
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:0 On active preferred
FC 6:2.1 210100e08bb27a58<->500508b30091aac9 vmhba1:0:4 On active preferred
All online!
Checking for death paths: esxcfg-mpath -l | grep -i death
[root@esxmeri01 /]#
None death paths!
Checking my HBA’s connected to the SAN: cd /vmfs/devices/disks and list: ls vmh*
vmhba1:0:1:0 vmhba1:0:2:1 vmhba1:0:4:0 vmhba1:0:5:1 vmhba1:0:7:0 vmhba2:0:0:1 vmhba2:0:0:3 vmhba2:0:0:6 vmhba2:0:0:9
vmhba1:0:1:1 vmhba1:0:3:0 vmhba1:0:4:1 vmhba1:0:6:0 vmhba1:0:7:1 vmhba2:0:0:10 vmhba2:0:0:4 vmhba2:0:0:7
vmhba1:0:2:0 vmhba1:0:3:1 vmhba1:0:5:0 vmhba1:0:6:1 vmhba2:0:0:0 vmhba2:0:0:2 vmhba2:0:0:5 vmhba2:0:0:8
All online!
Represent LUN’s from HP ACU to ESX hosts:
I unpresent and represented the LUN’s in the HP ACU to the hosts, did a rescan but no still no success
Reservation conflicts??
After some troubleshooting and trying to get my datastores online I found some information to point me in the right direction:
cd /var/log
# cat dmesg
resize_dma_pool: unknown device type 12
scsi2 (0,0,1) : RESERVATION CONFLICT
scsi2 (0,0,1) : RESERVATION CONFLICT
scsi2 (0,0,1) : RESERVATION CONFLICT
scsi2 (0,0,1) : RESERVATION CONFLICT
scsi2 (0,0,1) : RESERVATION CONFLICT
scsi2 (0,0,1) : RESERVATION CONFLICT
VMWARE: Device that would have been attached as scsi disk sda at scsi2, channel 0, id 0, lun 1
Has not been attached because this path could not complete a READ command eventhough a TUR worked.
result = 0x18 key = 0x0, asc = 0x0, ascq = 0x0
VMWARE: Device that would have been attached as scsi disk sda at scsi2, channel 0, id 0, lun 1
Has not been attached because it is a duplicate path or on a passive path
resize_dma_pool: unknown device type 12
VMWARE SCSI Id: Supported VPD pages for sda : 0x0 0x80 0x83 0xc0 0xb0 0xc1
VMWARE SCSI Id: Device id info for sda: 0x1 0x3 0x0 0x10 0x60 0x5 0x8 0xb3 0x0 0x93 0x89 0xc0 0x92 0x5 0xd2 0xd2 0x6d 0x13 0x0 0x15 0x2 0x3 0x0 0x20 0x36 0x30 0x30 0x35 0x30 0x38 0x42 0x33 0x30 0x30 0x39 0x33 0x38 0x39 0x43 0x30 0x39 0x32 0x30 0x35 0x44 0x32 0x44 0x32 0x36 0x44 0x31 0x33 0x30 0x30 0x31 0x35 0x1 0x6 0x0 0x4 0x0 0x0 0x0 0x2 0x1 0x14 0x0 0x4 0x0 0x0 0x0 0x1 0x1 0x15 0x0 0x4 0x0 0x0 0x0 0x1
VMWARE SCSI Id: Id for sda 0x60 0x05 0x08 0xb3 0x00 0x93 0x89 0xc0 0x92 0x05 0xd2 0xd2 0x6d 0x13 0x00 0x15 0x4d 0x53 0x41 0x20 0x56 0x4f
VMWARE: Unique Device attached as scsi disk sda at scsi2, channel 0, id 0, lun 2
Attached scsi disk sda at scsi2, channel 0, id 0, lun 2
resize_dma_pool: unknown device type 12
scsi2 (0,0,3) : RESERVATION CONFLICT
scsi2 (0,0,3) : RESERVATION CONFLICT
scsi2 (0,0,3) : RESERVATION CONFLICT
scsi2 (0,0,3) : RESERVATION CONFLICT
scsi2 (0,0,3) : RESERVATION CONFLICT
scsi2 (0,0,3) : RESERVATION CONFLICT
VMWARE: Device that would have been attached as scsi disk sdb at scsi2, channel 0, id 0, lun 3
Has not been attached because this path could not complete a READ command eventhough a TUR worked.
Checking the VMkering log file: tail -f /var/log/vmkernel
[Dec 21 10:44:25 esxmeri01 vmkernel: 0:19:06:51.306 cpu3:1037)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts
Dec 21 10:44:25 esxmeri01 vmkernel: 0:19:06:51.306 cpu3:1037)WARNING: SCSI: 255: status SCSI reservation conflict for vml.0200030000600508b3009389c079c769f1112400164d534120564f. residual R 919, CR 0, ER 3
Dec 21 10:44:26 esxmeri01 vmkernel: 0:19:06:52.243 cpu3:1037)SCSI: vm 1037: 109: Sync CR at 64
Dec 21 10:44:27 esxmeri01 vmkernel: 0:19:06:53.207 cpu3:1037)SCSI: vm 1037: 109: Sync CR at 48
Dec 21 10:44:28 esxmeri01 vmkernel: 0:19:06:54.124 cpu3:1037)SCSI: vm 1037: 109: Sync CR at 32
Dec 21 10:44:29 esxmeri01 vmkernel: 0:19:06:55.118 cpu3:1037)SCSI: vm 1037: 109: Sync CR at 16
Dec 21 10:44:30 esxmeri01 vmkernel: 0:19:06:56.098 cpu3:1037)SCSI: vm 1037: 109: Sync CR at 0
Dec 21 10:44:30 esxmeri01 vmkernel: 0:19:06:56.098 cpu3:1037)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts
In this case, LUN 12 is inaccessible in the ESX host cluster. Since it is listed in the output of step one, it was accessible at some point (a host reserved the LUN and never released it, possibly due to a SAN switch reboot in the middle of the reservation operation)
Solution:
Check pending reservations: esxcfg-info | egrep -B5 "s Reserved|Pending"
|—-Console Device…………………./dev/sda
|—-Devfs Path……………………../vmfs/devices/disks/vml.0200020000600508b3009389c09205d2d26d1300154d534120564f
|—-SCSI Level……………………..5
|—-Queue Depth…………………….32
|—-Is Pseudo………………………false
|—-Is Reserved…………………….false
|—-Pending Reservations…………….0
—
|—-Console Device…………………./dev/sdc
|—-Devfs Path……………………../vmfs/devices/disks/vml.0200060000600508b3009389c0df3c8fe88b3e001c4d534120564f
|—-SCSI Level……………………..5
|—-Queue Depth…………………….32
|—-Is Pseudo………………………false
|—-Is Reserved…………………….false
|—-Pending Reservations…………….1
—
Okay, as you can see disk “vml.0200060000600508b3009389c0df3c8fe88b3e001c4d534120564f “ has an Pending Reservation with status: 1.. you can solve this issue to do a LUN (connection) reset. You can do this on the fly, without losing any connection or data loss.
Now it’s time to do the LUN reset :
vmkfstools –lock lunreset /vmfs/devices/disks/vml.0200060000600508b3009389c0df3c8fe88b3e001c4d534120564f
I did a rescan on my HBA (each server) and voila… the datastores are back online again:
More information: VMware KB
About Author
I just added your blog site to my blogroll, I pray you would give some thought to doing the same.
Hi,
i have small question.
1.) what is the difference between # esxcfg-info -s | egrep -B16 “s Reserved|Pending” and esxcfg-info -s | grep -i -B 30 Pending(insert a backslash and then a space here)Reservations………………………….1
2.) what does ‘B 30 or B16’ means?
Note:- we have VMware ESX 4.0.
thnx in advance.
Valerian Crasto.
Jiiiihaaa this solved my problem, 100x many thanks
cheers for the actual article i’ve recently been on the lookout with regard to this kind of advice on the net for sum time proper now so numerous thanks ekecgfbcfecg
@Valerian Crasto
This is old but thought it is never to late to answer questions 🙂
________________________________________________
.) what is the difference between # esxcfg-info -s | egrep -B16 “s Reserved|Pending” and esxcfg-info -s | grep -i -B 30 Pending(insert a backslash and then a space here)Reservations………………………….1
2.) what does ‘B 30 or B16’ means?
________________________________________________
Basically the B16 or B30 have the same meaning… it is when you do a grep and you want either a number of line before with the B or after with the A of each matching occurence.
Ex…. grep -A 20 test
this will show the following 20 lines after the word test when it is found.
grep -B 20 test
will show the preceding 20 lines before the word test when it is found…