Hi all:

I'm running ESXi 5.5 to manage a testing platform.

I'm facing an important issue where my datastore storing all my virtual machines is not available anymore. It looks that I always get I/O error.

Even if I run the "rescan function", I don't see my datastore.

Disk is properly seen when I run "esxcfg-scsidevs -l"

naa.5000c50065953547

   Device Type: Direct-Access

   Size: 1907729 MB

   Display Name: ATA Serial Attached SCSI Disk (naa.5000c50065953547)

   Multipath Plugin: NMP

   Console Device: /vmfs/devices/disks/naa.5000c50065953547

   Devfs Path: /vmfs/devices/disks/naa.5000c50065953547

   Vendor: ATA       Model: ST2000DM001-1CH1  Revis: HP34

   SCSI Level: 6  Is Pseudo: false Status: degraded

   Is RDM Capable: true  Is Removable: false

   Is Local: false Is SSD: false

   Other Names:

      vml.02000000005000c50065953547535432303030

   VAAI Status: unknown

Any help would be very good to me as I need to back-up data ...

Thanks

Now before I get to far, I will say I am quite sure that the host I am doing this on IS NOT on the HCL.

(why I am running old hardware? - its a community organization that I volunteer for that relies on dontated hardware)

However please read on.

Upgraded an IBM x3500 and an x3650 (same generation - they are basically tower and rack mount versions of the same kit) from esxi 5.1 to 5.5.

(Exactly the same issue on both)

Local Storage - IBM Serveraid (aacraid)

I was a bit confused as the disk controllers and luns still show as visible (just no VMFS recognised no matter how many times I rescan).

(normally the luns don’t appear because of a driver not working)

I found I was able to get the luns to come up by doing

“Add Storage”

“Disk/LUN”

(Takes 30-60 seconds then shows the luns)

It shows the disk arrays, I then click next and I select “Mount disk and keep signature” ..

The lun then comes up fine and everything works.

However rebooting it gets lost again and I have to repeat the process to get the VMFS volumes to come back.

Possible related entries in vmkernel.log

2014-04-22T19:56:42.265Z cpu2:33418)module heap vmklnx_aacraid: creation succeeded. id = 0x410a640c1000

2014-04-22T19:56:42.265Z cpu2:33418)<6>Adaptec aacraid driver 1.2.1.30300

2014-04-22T19:56:42.265Z cpu2:33418)PCI: driver aacraid is looking for devices

2014-04-22T19:56:42.265Z cpu2:33418)DMA: 612: DMA Engine 'vmklnxpci-0:3:0.0' created using mapper 'DMANull'.

2014-04-22T19:56:42.265Z cpu2:33418)DMA: 612: DMA Engine 'vmklnxpci-0:3:0.0' created using mapper 'DMANull'.

2014-04-22T19:56:42.308Z cpu2:33418)IRQ: 540: 0x2b <aacraid> sharable, flags 0x10

2014-04-22T19:56:42.308Z cpu2:33418)VMK_VECTOR: 218: Registered handler for shared interrupt 0xff2b, flags 0x10

2014-04-22T19:56:43.984Z cpu2:33418)<6>aacraid0: kernel 5.2-0[17003] Jul 25 2011

2014-04-22T19:56:43.984Z cpu2:33418)<6>aacraid0: monitor 5.2-0[17003]

2014-04-22T19:56:43.984Z cpu2:33418)<6>aacraid0: bios 5.2-0[17003]

<6>aacraid0: serial

2014-04-22T19:56:44.000Z cpu2:33418)VMK_PCI: 395: Device 0000:03:00.0 name: vmhba1

2014-04-22T19:56:44.000Z cpu2:33418)DMA: 612: DMA Engine 'vmhba1' created using mapper 'DMANull'.

2014-04-22T19:56:44.000Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C0:T0:L0': Vendor: 'ServeRA '  Model: 'SAS1            ' Rev: 'V1.0'

2014-04-22T19:56:44.000Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C0:T0:L0': Type: 0x0, ANSI rev: 2, TPGS: 0 (none)

2014-04-22T19:56:44.000Z cpu2:33418)ScsiUid: 273: Path 'vmhba1:C0:T0:L0' does not support VPD Device Id page.

2014-04-22T19:56:44.000Z cpu2:33418)ScsiScan: 1105: Path 'vmhba1:C0:T0:L0' : No standard UID: Failure. ANSI version 'SCSI-2' (0x2).

2014-04-22T19:56:44.000Z cpu2:33418)VMWARE SCSI Id: Id for vmhba1:C0:T0:L0

0x39 0x62 0x34 0x63 0x61 0x30 0x61 0x63 0x53 0x41 0x53 0x31 0x20 0x20

2014-04-22T19:56:44.000Z cpu2:33418)ScsiScan: 1503: Add path: vmhba1:C0:T0:L0

2014-04-22T19:56:44.000Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C0:T1:L0': Vendor: 'ServeRA '  Model: 'SATA1           '  Rev: 'V1.0'

2014-04-22T19:56:44.000Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C0:T1:L0': Type: 0x0, ANSI rev: 2, TPGS: 0 (none)

2014-04-22T19:56:44.001Z cpu2:33418)ScsiUid: 273: Path 'vmhba1:C0:T1:L0' does not support VPD Device Id page.

2014-04-22T19:56:44.001Z cpu2:33418)ScsiScan: 1105: Path 'vmhba1:C0:T1:L0' : No standard UID: Failure. ANSI version 'SCSI-2' (0x2).

2014-04-22T19:56:44.001Z cpu2:33418)VMWARE SCSI Id: Id for vmhba1:C0:T1:L0

0x39 0x37 0x39 0x38 0x61 0x30 0x61 0x63 0x53 0x41 0x54 0x41 0x31 0x20

2014-04-22T19:56:44.001Z cpu2:33418)ScsiScan: 1503: Add path: vmhba1:C0:T1:L0

2014-04-22T19:56:44.018Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T0:L0': Vendor: 'IBM-ESXS'  Model: 'ST3300655SS     '  Rev: 'BA2D'

2014-04-22T19:56:44.018Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T0:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.020Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T0:L0 : Not found

2014-04-22T19:56:44.021Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T1:L0': Vendor: 'IBM-ESXS'  Model: 'ST3300655SS     '  Rev: 'BA2D'

2014-04-22T19:56:44.021Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T1:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.023Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T1:L0 : Not found

2014-04-22T19:56:44.025Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T2:L0': Vendor: 'IBM-ESXS'  Model: 'ST3300655SS     '  Rev: 'BA2D'

2014-04-22T19:56:44.025Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T2:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.027Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T2:L0 : Not found

2014-04-22T19:56:44.029Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T3:L0': Vendor: 'IBM-ESXS'  Model: 'VPBA300C3ETS11 N' Rev: 'A496'

2014-04-22T19:56:44.029Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T3:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.032Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T3:L0 : Not found

2014-04-22T19:56:44.034Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T4:L0': Vendor: 'HUA72202'  Model: '0ALA330         '  Rev: 'A34H'

2014-04-22T19:56:44.034Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T4:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.039Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T4:L0 : Not found

2014-04-22T19:56:44.041Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T5:L0': Vendor: 'HUA72202'  Model: '0ALA330         '  Rev: 'A34H'

2014-04-22T19:56:44.041Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T5:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.047Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T5:L0 : Not found

2014-04-22T19:56:44.049Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T6:L0': Vendor: 'HUA72202'  Model: '0ALA330 '  Rev: 'A34H'

2014-04-22T19:56:44.049Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T6:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.054Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T6:L0 : Not found

2014-04-22T19:56:44.056Z cpu2:33418)ScsiScan: 976: Path 'vmhba1:C1:T7:L0': Vendor: 'HUA72202'  Model: '0ALA330         '  Rev: 'A3GA'

2014-04-22T19:56:44.056Z cpu2:33418)ScsiScan: 979: Path 'vmhba1:C1:T7:L0': Type: 0x0, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.061Z cpu2:33418)WARNING: ScsiScan: 1408: Failed to add path vmhba1:C1:T7:L0 : Not found

2014-04-22T19:56:44.844Z cpu3:33418)ScsiScan: 976: Path 'vmhba1:C3:T0:L0': Vendor: 'IBM     '  Model: 'SAS SES-2 DEVICE'  Rev: '1.11'

2014-04-22T19:56:44.844Z cpu3:33418)ScsiScan: 979: Path 'vmhba1:C3:T0:L0': Type: 0xd, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.845Z cpu3:33418)ScsiScan: 1503: Add path: vmhba1:C3:T0:L0

2014-04-22T19:56:44.874Z cpu3:33418)ScsiScan: 976: Path 'vmhba1:C3:T1:L0': Vendor: 'IBM     '  Model: 'SAS SES-2 DEVICE'  Rev: '1.11'

2014-04-22T19:56:44.874Z cpu3:33418)ScsiScan: 979: Path 'vmhba1:C3:T1:L0': Type: 0xd, ANSI rev: 5, TPGS: 0 (none)

2014-04-22T19:56:44.874Z cpu3:33418)ScsiScan: 1503: Add path: vmhba1:C3:T1:L0

2014-04-22T19:56:44.903Z cpu3:33418)PCI: driver aacraid claimed device 0000:03:00.0

2014-04-22T19:56:44.903Z cpu3:33418)PCI: driver aacraid claimed 1 device

2014-04-22T19:56:44.903Z cpu3:33418)ScsiNpiv: 1510: GetInfo for adapter vmhba1, [0x4108991c4400], max_vports=0, vports_inuse=0, linktype=0, state=0, failreason=0, sts=bad0020

2014-04-22T19:56:44.903Z cpu3:33418)Mod: 4780: Initialization of aacraid succeeded with module ID 4174.

2014-04-22T19:56:44.903Z cpu3:33418)aacraid loaded successfully.

2014-04-22T19:57:07.090Z cpu6:33204)ScsiPath: 4695: Plugin 'NMP' claimed path 'vmhba32:C0:T0:L0'

2014-04-22T19:57:07.090Z cpu6:33204)ScsiPath: 4695: Plugin 'NMP' claimed path 'vmhba0:C0:T0:L0'

2014-04-22T19:57:07.091Z cpu6:33204)ScsiPath: 4695: Plugin 'NMP' claimed path 'vmhba1:C3:T1:L0'

2014-04-22T19:57:07.091Z cpu6:33204)ScsiPath: 4695: Plugin 'NMP' claimed path 'vmhba1:C3:T0:L0'

2014-04-22T19:57:07.091Z cpu6:33204)ScsiPath: 4695: Plugin 'NMP' claimed path 'vmhba1:C0:T1:L0'

2014-04-22T19:57:07.091Z cpu6:33204)ScsiPath: 4695: Plugin 'NMP' claimed path 'vmhba1:C0:T0:L0'

2014-04-22T19:57:07.091Z cpu6:33204)ScsiUid: 273: Path 'vmhba1:C0:T0:L0' does not support VPD Device Id page.

2014-04-22T19:57:07.092Z cpu6:33204)VMWARE SCSI Id: Id for vmhba1:C0:T0:L0

0x39 0x62 0x34 0x63 0x61 0x30 0x61 0x63 0x53 0x41 0x53 0x31 0x20 0x20

2014-04-22T19:57:07.092Z cpu6:33204)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:479: Changing active path from NONE to vmhba1:C0:T0:L0 for device "Unregistered Device".

2014-04-22T19:57:07.092Z cpu6:33204)StorageApdHandler: 698: APD Handle  Created with lock[StorageApd0x410899]

2014-04-22T19:57:07.092Z cpu6:33204)ScsiEvents: 501: Event Subsystem: Device Events, Created!

2014-04-22T19:57:07.092Z cpu6:33204)VMWARE SCSI Id: Id for vmhba1:C0:T0:L0

0x39 0x62 0x34 0x63 0x61 0x30 0x61 0x63 0x53 0x41 0x53 0x31 0x20 0x20

2014-04-22T19:57:07.092Z cpu6:33204)ScsiDeviceIO: 7419: Get VPD 83 Inquiry for device "mpx.vmhba1:C0:T0:L0" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.092Z cpu6:33204)ScsiDeviceIO: 7455: Get VPD 86 Inquiry for device "mpx.vmhba1:C0:T0:L0" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.092Z cpu7:33204)ScsiDeviceIO: 6169: Could not detect setting of QErr for device mpx.vmhba1:C0:T0:L0. Error Failure.

2014-04-22T19:57:07.114Z cpu2:33204)FSS: 5092: No FS driver claimed device 'mpx.vmhba1:C0:T0:L0:1': Not supported

2014-04-22T19:57:07.114Z cpu2:33204)ScsiDevice: 3443: Successfully registered device "mpx.vmhba1:C0:T0:L0" from plugin "NMP" of type 0

2014-04-22T19:57:07.114Z cpu2:33204)ScsiUid: 273: Path 'vmhba1:C0:T1:L0' does not support VPD Device Id page.

2014-04-22T19:57:07.114Z cpu2:33204)VMWARE SCSI Id: Id for vmhba1:C0:T1:L0

0x39 0x37 0x39 0x38 0x61 0x30 0x61 0x63 0x53 0x41 0x54 0x41 0x31 0x20

2014-04-22T19:57:07.114Z cpu2:33204)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:479: Changing active path from NONE to vmhba1:C0:T1:L0 for device "Unregistered Device".

2014-04-22T19:57:07.114Z cpu2:33204)StorageApdHandler: 698: APD Handle  Created with lock[StorageApd0x410899]

2014-04-22T19:57:07.114Z cpu2:33204)ScsiEvents: 501: Event Subsystem: Device Events, Created!

2014-04-22T19:57:07.114Z cpu2:33204)VMWARE SCSI Id: Id for vmhba1:C0:T1:L0

0x39 0x37 0x39 0x38 0x61 0x30 0x61 0x63 0x53 0x41 0x54 0x41 0x31 0x20

2014-04-22T19:57:07.114Z cpu2:33204)ScsiDeviceIO: 7419: Get VPD 83 Inquiry for device "mpx.vmhba1:C0:T1:L0" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.114Z cpu2:33204)ScsiDeviceIO: 7455: Get VPD 86 Inquiry for device "mpx.vmhba1:C0:T1:L0" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.115Z cpu2:33204)ScsiDeviceIO: 6169: Could not detect setting of QErr for device mpx.vmhba1:C0:T1:L0. Error Failure.

2014-04-22T19:57:07.154Z cpu2:33204)FSS: 5092: No FS driver claimed device 'mpx.vmhba1:C0:T1:L0:1': Not supported

2014-04-22T19:57:07.154Z cpu2:33204)ScsiDevice: 3443: Successfully registered device "mpx.vmhba1:C0:T1:L0" from plugin "NMP" of type 0

2014-04-22T19:57:07.156Z cpu2:33204)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:479: Changing active path from NONE to vmhba1:C3:T0:L0 for device "Unregistered Device".

2014-04-22T19:57:07.156Z cpu2:33204)StorageApdHandler: 698: APD Handle  Created with lock[StorageApd0x410899]

2014-04-22T19:57:07.156Z cpu2:33204)ScsiEvents: 501: Event Subsystem: Device Events, Created!

2014-04-22T19:57:07.156Z cpu2:33204)VMWARE SCSI Id: Id for vmhba1:C3:T0:L0

0x50 0x05 0x07 0x6c 0x07 0xbf 0xe2 0x79 0x53 0x41 0x53 0x20 0x53 0x45

2014-04-22T19:57:07.156Z cpu2:33204)ScsiDeviceIO: 7409: Get VPD 80 Inquiry for device "naa.5005076c07bfe279" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.156Z cpu2:33204)ScsiDeviceIO: 7455: Get VPD 86 Inquiry for device "naa.5005076c07bfe279" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.166Z cpu3:32792)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808916c0, 0) to dev "naa.5005076c07bfe279" on path "vmhba1:C3:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2014-04-22T19:57:07.166Z cpu3:32792)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5005076c07bfe279" state in doubt; requested fast path state update...

2014-04-22T19:57:07.166Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x24 from world 0 to dev "naa.5005076c07bfe279" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.187Z cpu3:32792)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808916c0, 0) to dev "naa.5005076c07bfe279" on path "vmhba1:C3:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2014-04-22T19:57:07.207Z cpu2:33204)ScsiDeviceIO: 6169: Could not detect setting of QErr for device naa.5005076c07bfe279. Error Storage initiator error.

2014-04-22T19:57:07.217Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x25 from world 0 to dev "naa.5005076c07bfe279" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.253Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x26 from world 0 to dev "naa.5005076c07bfe279" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.289Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x27 from world 0 to dev "naa.5005076c07bfe279" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.317Z cpu3:32792)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808916c0, 0) to dev "naa.5005076c07bfe279" on path "vmhba1:C3:T0:L0" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2014-04-22T19:57:07.317Z cpu2:33204)ScsiDevice: 3443: Successfully registered device "naa.5005076c07bfe279" from plugin "NMP" of type 13

2014-04-22T19:57:07.318Z cpu2:33204)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:479: Changing active path from NONE to vmhba1:C3:T1:L0 for device "Unregistered Device".

2014-04-22T19:57:07.318Z cpu2:33204)StorageApdHandler: 698: APD Handle  Created with lock[StorageApd0x410899]

2014-04-22T19:57:07.318Z cpu2:33204)ScsiEvents: 501: Event Subsystem: Device Events, Created!

2014-04-22T19:57:07.318Z cpu2:33204)VMWARE SCSI Id: Id for vmhba1:C3:T1:L0

0x50 0x05 0x07 0x6c 0x07 0xbf 0xe2 0x7b 0x53 0x41 0x53 0x20 0x53 0x45

2014-04-22T19:57:07.318Z cpu2:33204)ScsiDeviceIO: 7409: Get VPD 80 Inquiry for device "naa.5005076c07bfe27b" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.318Z cpu2:33204)ScsiDeviceIO: 7455: Get VPD 86 Inquiry for device "naa.5005076c07bfe27b" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.328Z cpu3:32792)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808916c0, 0) to dev "naa.5005076c07bfe27b" on path "vmhba1:C3:T1:L0" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2014-04-22T19:57:07.328Z cpu3:32792)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5005076c07bfe27b" state in doubt; requested fast path state update...

2014-04-22T19:57:07.328Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x28 from world 0 to dev "naa.5005076c07bfe27b" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.350Z cpu3:32792)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808916c0, 0) to dev "naa.5005076c07bfe27b" on path "vmhba1:C3:T1:L0" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2014-04-22T19:57:07.371Z cpu2:33204)ScsiDeviceIO: 6169: Could not detect setting of QErr for device naa.5005076c07bfe27b. Error Storage initiator error.

2014-04-22T19:57:07.381Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x29 from world 0 to dev "naa.5005076c07bfe27b" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.417Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x2a from world 0 to dev "naa.5005076c07bfe27b" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.453Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e808916c0) 0x1a, CmdSN 0x2b from world 0 to dev "naa.5005076c07bfe27b" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2014-04-22T19:57:07.479Z cpu3:32792)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808916c0, 0) to dev "naa.5005076c07bfe27b" on path "vmhba1:C3:T1:L0" Failed: H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

2014-04-22T19:57:07.479Z cpu2:33204)ScsiDevice: 3443: Successfully registered device "naa.5005076c07bfe27b" from plugin "NMP" of type 13

2014-04-22T19:57:07.501Z cpu2:33204)ScsiUid: 273: Path 'vmhba0:C0:T0:L0' does not support VPD Device Id page.

2014-04-22T19:57:07.510Z cpu2:33204)VMWARE SCSI Id: Could not get disk id for vmhba0:C0:T0:L0

2014-04-22T19:57:07.510Z cpu2:33204)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:479: Changing active path from NONE to vmhba0:C0:T0:L0 for device "Unregistered Device".

2014-04-22T19:57:07.511Z cpu2:33204)StorageApdHandler: 698: APD Handle  Created with lock[StorageApd0x410899]

2014-04-22T19:57:07.511Z cpu2:33204)ScsiEvents: 501: Event Subsystem: Device Events, Created!

2014-04-22T19:57:07.520Z cpu2:33204)VMWARE SCSI Id: Could not get disk id for vmhba0:C0:T0:L0

2014-04-22T19:57:07.520Z cpu2:33204)ScsiDeviceIO: 7409: Get VPD 80 Inquiry for device "mpx.vmhba0:C0:T0:L0" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.520Z cpu2:33204)ScsiDeviceIO: 7419: Get VPD 83 Inquiry for device "mpx.vmhba0:C0:T0:L0" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.520Z cpu2:33204)ScsiDeviceIO: 7455: Get VPD 86 Inquiry for device "mpx.vmhba0:C0:T0:L0" from Plugin "NMP" failed. Not supported

2014-04-22T19:57:07.520Z cpu2:33204)ScsiDeviceIO: 6169: Could not detect setting of QErr for device mpx.vmhba0:C0:T0:L0. Error Medium not found.

2014-04-22T19:57:07.520Z cpu3:33204)ScsiDevice: 3443: Successfully registered device "mpx.vmhba0:C0:T0:L0" from plugin "NMP" of type 5

2014-04-22T19:57:07.521Z cpu4:33366)usb-storage: detected SCSI revision number 6 on vmhba32

2014-04-22T19:57:07.521Z cpu4:33366)usb-storage: patching inquiry data to change SCSI revision number from 6 to 2 on vmhba32

2014-04-22T19:57:07.521Z cpu4:33366)usb-storage: patching inquiry data to change removable media bit from 'off' to 'on' on vmhba32

2014-04-22T19:57:13.043Z cpu3:32792)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5005076c07bfe279" state in doubt; requested fast path state update...

2014-04-22T19:57:13.043Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e80898f40) 0x1a, CmdSN 0xd0 from world 0 to dev "naa.5005076c07bfe279" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.

2014-04-22T19:57:13.081Z cpu3:32792)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.5005076c07bfe27b" state in doubt; requested fast path state update...

2014-04-22T19:57:13.081Z cpu3:32792)ScsiDeviceIO: 2337: Cmd(0x412e80898f40) 0x1a, CmdSN 0xd1 from world 0 to dev "naa.5005076c07bfe27b" failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.

2014-04-22T19:57:13.123Z cpu2:33204)LVM: 8355: Device mpx.vmhba1:C0:T1:L0:1 detected to be a snapshot:

2014-04-22T19:57:13.123Z cpu2:33204)LVM: 8362: queried disk ID: <type 1, len 14, lun 0, devType 0, scsi 0, h(id) 16746273972281211240>

2014-04-22T19:57:13.123Z cpu2:33204)LVM: 8369: on-disk disk ID: <type 0, len 10, lun 0, devType 0, scsi 0, h(id) 1308713170419776745>

2014-04-22T19:57:13.124Z cpu2:33204)LVM: 8355: Device mpx.vmhba1:C0:T0:L0:1 detected to be a snapshot:

2014-04-22T19:57:13.124Z cpu2:33204)LVM: 8362: queried disk ID: <type 1, len 14, lun 0, devType 0, scsi 0, h(id) 15010766293529110797>

2014-04-22T19:57:13.124Z cpu2:33204)LVM: 8369: on-disk disk ID: <type 0, len 10, lun 0, devType 0, scsi 0, h(id) 13427631733060988540>

2014-04-22T19:57:13.131Z cpu2:33204)FSS: 5092: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported

2014-04-22T19:57:13.155Z cpu2:33204)Vol3: 714: Couldn't read volume header from control: Not supported

2014-04-22T19:57:13.155Z cpu2:33204)Vol3: 714: Couldn't read volume header from control: Not supported

2014-04-22T19:57:13.155Z cpu2:33204)FSS: 5092: No FS driver claimed device 'control': Not supported

2014-04-22T19:57:13.202Z cpu2:33204)VC: 2059: Device rescan time 101 msec (total number of devices 5)

2014-04-22T19:57:13.202Z cpu2:33204)VC: 2062: Filesystem probe time 70 msec (devices probed 5 of 5)

2014-04-22T19:57:13.202Z cpu2:33204)VC: 2064: Refresh open volume time 0 msec

2014-04-22T19:57:14.826Z cpu3:32851)LVM: 13089: One or more LVM devices have been discovered.

Starting up services2014-04-22T19:57:16.014Z cpu0:32895)Tcpip_Vmk: 123: nd6_rs_timer: no routers on 'vmk0'. Giving up

2014-04-22T19:57:16.531Z cpu1:33651)VisorFSRam: 634: vsantraces with (0,300,0,0,755)

2014-04-22T19:57:16.531Z cpu1:33651)FSS: 6895: Mounting fs visorfs (4108aa584010) with -o 0,300,0,0,1,0755,vsantraces on file descriptor 4108af971220

Possible related entries in syslog.log

2014-04-15T04:50:20Z jumpstart: Storage Info: Unable to Mount VMFS volume with UUID 522a475e-80e96251-5f62-00215e4e123e. Sysinfo error on operation returned status : Bad parameter count. Please see the VMkernel log for detailed error information

2014-04-15T04:50:20Z jumpstart: Storage Info: Unable to Mount VMFS volume with UUID 522a4783-5bbdaecf-9237-00215e4e123e. Sysinfo error on operation returned status : Bad parameter count. Please see the VMkernel log for detailed error information

Anyone have any helpful thoughts ?

Thankyou!

Good day,

I'm seeing (rare) excessive UDP packet propagation delays for incoming traffic with Debian wheezy kernel 3.2.0-p4 (default kernel of stable debian release) and vmxnet3 on ESXi 5.1.

We have a VM that is driving digitizers that send data with about 40-60MB/s. After some physical NIC tuning we see zero packet drop. However over a period of 12hours I see 180 cases (i.e. on average every 4 minutes) where a UDP packet was delivered with rather extreme delay. We use a blocking select() with timeout followed by recv() to read from the UDP socket.

Observations:

• With a select timeout of 24ms we often get timeout overshot of up to 30ms with the default kernel - i.e. the select call does not return after 24ms but after 30, 40 even 54ms.
• Every single time we see timeout overshot (of up to 30ms!), less than 0.1ms later the next packet arrives.
• The incoming packets have sequence numbers, so I could verify that it is not packet loss.

I've compiled a custom kernel with

# CONFIG_RCU_FAST_NO_HZ is not set

CONFIG_NO_HZ=y

CONFIG_HZ_1000=y

CONFIG_HZ=1000

which drastically reduces the timeout overshot. So my first thought is that this is an RCU issue in/with the driver. However what really puzzles me is that the data is always there 0.1ms later. So it almost seems like the driver is missing to "raise an interrupt" on incoming data from time to time.

Any help appreciated.

Hello,

The ESX have a total of 70Gb of physical memory.

I have allocated 24Gb of memory on a VM, according to the guest OS task manager, it only use 2Gb, but Vsphere tells me that the VM consumed all the allocated memory.

I have setup a 12Gb reservation, so I would have understand if the consumed memory was 12Gb, but it is not, it is 24Gb. It will be easier to explain with some screenshot:

This is what I see in Vsphere:

And this is what I see in the guest OS task manager (1.98Gb used memory) :

I tried to put a limit to 12Gb, but when I do so, private memory go to 12Gb, and the other half of the allocated memory is balloon !

It seems that VMware really thinks that my VM is using all of his allocated memory.

I saw someone with the same issue : https://communities.vmware.com/thread/473652 but I didn't understand what was his solution.

Thank you. (And sorry if I made some writting mistakes, English isn't my native tongue)

Hello,

From my experience, when using vMotion, I usually see a missed ping or none at all at the exact moment a virtual machine cuts over to the destination host.

However, I'm now observing a network disconnect that can be over ten seconds during the vMotion cutover, which is causing issues.

This is happening mostly on HP Proliant DL580s and DL980s with dual 10Gb NICs for vMotion and dual 10Gb NICs for VM traffic. Storage is FC with 2TB LUNs. ESX is 5.1 Update 1 on vCenter 5.1.

The affected VMs are usually between 24 and 64Gb of memory with 16 vCPUs.

I'm also observing long ping timeouts during a Storage vMotion, once again right at the end of the operation.

So far, VMware support have not been able to find a solution to this issue but we are still investigating.

Has anyone observed this issue in their environment?

Let me know if you need additional information.

Thank you.

Hello,

Can someone please help me determine what this error means.

I upgraded the firmware and driver on my 10GB Emulex cards and I now see this error in my vmkwarning.log file.

Please note that the error is temporary. when I tail the file, it only shows the error came up after the reboot

Hi all,

I am creating two sql cluster nodes in VMware host cluster.

I created two VMs and installed MS failover cluster and added these two nodes.

I shared .vmdk disk (in datastore), with two machines and it is appearing fine. I followed these steps:

https://communities.vmware.com/blogs/Abhilash_hb/2013/08/25/clustering-using-sharing-of-vmdks-between-virtual-machines

then after adding disk in failover cluster I run cluster validation test.

but found following errors:

validate disk failover : cancelled

Validate SCSI-3 persistent reservation : Failed

these two VMs were on different  VMware host so I moved machines on one host then above errors gone but found following error:

Validate storage spaces persistent reservation : warning (yellow)

is there any compatibility issue?

also when I tried to migrate VM from one host to another I found following errors:

virtual machine is configured to use a device that prevents the operation: Device 'SCSI controller 1' is SCSI controller engaged in bus- sharing.

virtual machine is configured to use a device that prevents the operation : Device 'SCSI controller 2' is SCSI controller engaged in bus - sharing.

I have selected physical as bus sharing mode. in SCSI controller setting.

so does it means Microsoft VM cluster in VMware must be on same host ?

so there are two questions about:

Validate storage spaces persistent reservation ; warning

so does it means Microsoft VM cluster in VMware must be on same host and migration is not supported?

I am using ESXi 5.5

Dell MD 3220 SAS storage

windows 2012 R2 standard (VM cluster machines).

Thank you,

PARKAR

UAE

We recently updated to ESXi 5.5.   As of a week ago we cannot login into the Host using vsphere Client.  I have search for a solution but all appear to fix a problem with Server 2003 / Internet Explorer 8.0. 

The error I receive is:  

vSphere Client could not connect to "x.x.x.x".

An unknown connection error occurred. (The server could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable.))

I tried to log into the host with Vsphere client from 2 different machines. Important:  I only have problems logging into 1 of  2 ESXi 5.5 hosts.

I am a newbie-----Thanks for any help.

I am getting the below error when attempting to install a enic driver manually.  I have tried with the offline bundle .zip as well as the .vib file.  There is not an image profile on this host, can anyone give any insight as to what might be going on here?

[DependencyError]

VIB Cisco_bootbank_net-enic_2.1.2.42-1OEM.550.0.0.1198611 requires vmkapi_2_2_0_0, but the requirement cannot be satisfied within the ImageProfile.

VIB Cisco_bootbank_net-enic_2.1.2.42-1OEM.550.0.0.1198611 requires com.vmware.driverAPI-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.

Help, our VM's are running at a crawl ever since a power outage a few days ago. All hardware appears to be in good shape.

Host is DL385 G7, Dual 12 core AMD, 48GB memory, Primary Datastore is local storage (4) SAS 600GB run in RAID 10 for a total of 1.2 GB

Where do I even begin to start looking?

4 Critical VM's:

Domain Controller - Server 2008

SQL Server - Server 2008

Terminal Server - Server 2008 R2

Mail Gateway - Windows 7

3 other various OS VM's:

Windows XP

Server 2008

Linux Based UPS power management virtual appliance.

I powered off these 3 for now hoping to free up resources (doesn't help any).

CPU and memory usage is nothing more than normal. Just that the VM's are running so slow to the point they are nearly unusable (screens refresh block by block)

Every client computer is a thin-client and they often loose communication for 10-20 seconds at a time, I'm sure due to time outs.

If screen captures of the VM folder contents or anything else would help, let me know.

Hello,

after installing the Dell OMSA 6.5 A02 VIB (esxcli software vib install -d /.../omsa.zip) I can no longer scan the Host with Update Manager. I get the error message:

The host returns esxupdate error code:99. An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details.

After looking in /var/log/esxupdate.log I see the following lines, which could be the Reason why the scanning of the Host fails:

2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR: An unexpected exception was caught:
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 216, in main
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR:     cmd.Run()
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 106, in Run
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 78, in DownloadMetadatas
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/VibCollection.py", line 60, in __add__
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/VibCollection.py", line 77, in AddVib
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/Vib.py", line 625, in MergeVib
2011-12-14T12:40:39Z esxupdate: esxupdate: ERROR: ValueError: Cannot merge VIBs Dell_bootbank_OpenManage_6.5-0000, Dell_bootbank_OpenManage_6.5-0000 with unequal payloads attributes
2011-12-14T12:40:39Z esxupdate: esxupdate: DEBUG: <<<

Now I thought "Ok lets remove the VIB ("esxcli software vib remove OpenManage:6.5-000" afterwards "reboot") then everything will work again" but sadly it didn´t . The VIB is no longer listed (esxcli software vib list) but the error and log entrys stayed the same.

Someone else having the same or similar problem or know a way to get out of this mess?

Best regards,

Bjoern

Hello,

Here is my problem.  I have an ESXI 5.1 server with 3 virtual servers.  1 is Vcenter  another is a server for Vranger, and a production server.  The raid controller and 2 disk in the raid died.  I have a backup of the virtual servers.  After I rebuild the ESXI 5.1 server what will the impact be for the VCENTER server.

Thanks

I am constantly getting my nexus 1000v vsphere network distributed switch reconfigured under the tasks initiated by

Cisco_Nexus_1000v_xxxxxxxx

any idea what this is?

Can someone verify this setup is correct, and make suggestions on improvements?

Hardware:

2x HP DL380p Gen8 Servers

1 CPU each, 10 Cores, 2.2 GHz

32 GB Ram each

8x Gbps NICs

2x HP Procurve 2920 24-port Gigabit Switches, non-stacking, setup identical

Switches are trunked together, VLANs can pass through the trunk

Ports 13-20 are set to VLAN 10 untagged, and have jumbo frames enabled for Storage Network

HP MSA 2040 SAN

4x SAS 450GB 15k rpm RAID-5 volume "production"

4x SAS 3TB 7200 rpm RAID-5 volume "storage"

iSCSI interfaces:

Controller1:

192.168.130.10

192.168.131.10

192.168.132.10

192.168.133.10

Controller2:

192.168.130.11

192.168.131.11

192.168.132.11

192.168.133.11

Software:

ESXi 5.5 Essentials Plus

Cluster with 2 Hosts, HA is enabled

Host1 Networking:

vSwitch0:  VM Network & Management Network

NICs 0,4  (Active, Active) load balancing port ID - one cable going to each HP Procurve 2920 Switch

vSwitch1:  iSCSI VMkernel Ports

NICs 1,2,5,6 - each iSCSI is Port Binding to a specific NIC, 9000 MTU, two cables going to each HP Procurve 2920 Switch

192.168.130.20

192.168.131.20

192.168.132.20

192.168.133.20

Host2 Networking:

vSwitch0:  VM Network & Management Network

NICs 0,4  (Active, Active) load balancing port ID - one cable going to each HP Procurve 2920 Switch

vSwitch1:  iSCSI VMkernel Ports

NICs 1,2,5,6 - each iSCSI is Port Binding to a specific NIC, 9000 MTU, two cables going to each HP Procurve 2920 Switch

192.168.130.21

192.168.131.21

192.168.132.21

192.168.133.21

Each datastore is set to Round Robin path selection.

Hello,

I'm encountering a problem where VMs are loosing network connectivity right before cutover to new host when performing VMotion. 

My infrastructure is on HP Blades Gen 8  running Virtual Connect 3.70 connected to Nexus 5000 switches.  ESXi 5.1 Update 2 with VDs for the cluster.  VMotion is being performed on its own VLAN to segregate the traffic.  It was also happening on 5.1 Update 1.  I'm at a loss here.  Has anyone seen this before?

Migration of the VM completes but bot before some network timeouts.  I was made aware of the situation and watched a ping on the server while it was being migrated and it happens with all my VMs on any of the 5 clusters I have.

Any ideas?

Hi all

The Systems:

Host:          HP ProLiant DL380p Gen8 with ESXi 5.5 U1 (1623387)

Storage:     NetApp FAS2240 with OnTAP 8.2.1

If I want toadd anNFSdatastore, I get the followingerror message:

Call "HostDatastoreSystem.CreateNasDatastore" for
object "ha-datastoresystem" on ESXi "HOST IP" failed.

NFS mount IPADDRESS:/vol/vol_nfs_ds01/q_nfs_ds01 failed: The
NFS server does not support MOUNT version 3 over TCP.

An unknown error has occurred.

Does anyone havea solution?

Kind regards

Martin

i am not able to access admin portal of vcloud with administrator account however i can use same user account to connect with vcloud cell via ssh. anyone who can help me to resolve this issue.

Hi team,

I am trying to gather the hpacucli information using get-esxcli

• Connecting Directly to ESXi Host in PowerCLI with root credentials
• $esxcli = Get-VMHost | Get-EsxCli
• $esxcli.hpacucli.cmd("controller all show status")

Unable to cast object of type 'System.String' to type 'System.Collections.Gener

ic.Dictionary`2[System.String,System.Object]'.

At line:1 char:21

+ $esxcli.hpacucli.cmd <<<< ("$command")

    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException

    + FullyQualifiedErrorId : MethodInvocationException

Not sure where i am doing wrong. Please help me.

I ran update 3 for ESXi 5.0 this morning (it was previously on update 2) and it said it completed successfully. After a reboot I connected to the server and it says there is no persistent storage. My datastores are no longer listed. I suspect it might be a driver issue or something of the sort. Can someone point me in the right direction?

EDIT: Here is a screen shot of what I see:

Thanks

Hello everyone,

I've started using Vmware ESXi 5.0 recently. I've enacted a host with following version. I'm having difficulty creating Windows Server 2012 R2 based VMs where upon booting it blows up and restarts continuously. Then I found this  (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006859)KB article.

Does this patch applies to this installation of VMware ESXi 5.0 I'm currently? Please help. Thanks.

Regards,

Shahzad.