Hello,
I understand this server is still not on the HCL, and to make things clear we are just using this for our Lab at the moment and using it as a learning platform for Vsphere 5, the system was running without any issues for 2 months til 3 weeks ago, when I had an issue I guess with the local storage (we are not using any SAN or NAS on the test system at the moment), this server is using the LSI 2108 megaraid card btw.
anyway, the issue is , every week (exactly 1 week), the VM's & host becomre extremely slow to unresponsive at times, you can still ping everything, but other than that, nothing is responding, When I try to access the host via the vsphere client, its stuck at loading inventory, when I access the DCUI , its terribly slow as well, the system logs load forever, I can't even soft reset everything (it just becomes stuck on the DCUI, nothing happens). only a hard reset solves this.
Having been working with vmware/virtualization for some time now, I had my instincts pointing at the storage ( correct me if im mistaken ), and checking the vmkernel log , Im seeing errors such as
2012-08-26T18:04:53.393Z cpu1:445134)<5>0 :: megasas: RESET -56651393 cmd=28 retries=0
2012-08-26T18:04:53.393Z cpu1:445134)megaraid_sas: HBA reset handler invoked without an internal reset condition.
2012-08-26T18:04:53.393Z cpu6:2054)WARNING: ScsiDeviceIO: 1218: Device naa.6003048003445f001789a91805f3b48d performance has deteriorated. I/O latency increased from average value of 8552 microseconds to 3848124 microseconds.
2012-08-26T18:04:53.393Z cpu3:2067)WARNING: LinScsi: SCSILinuxQueueCommand:1175:queuecommand failed with status = 0x1055 Host Busy vmhba2:2:0:0 (driver name: LSI Logic SAS based MegaRAID driver) - Message repeated 59979 times
2012-08-26T18:04:54.405Z cpu1:445134)<7>megaraid_sas: megasas_wait_for_outstanding: line 1875: AFTER HBA reset handler invoked without an internal reset condition: took 1 seconds. Max is 180.
2012-08-26T18:04:54.405Z cpu1:445134)megaraid_sas: no more pending commands remain after reset handling.
2012-08-26T18:04:54.405Z cpu1:445134)<5>megasas: reset successful
2012-08-26T18:05:03.393Z cpu3:2863)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3559424 gen 29 stampUS 661702864527 uuid 50304c73-629ecc56-8644-002590521fbd jrnl <FB 6049> drv 14.54] on vol 'datastore1'
(I can upload the whole log if needed), I just want to be sure, can this be a hardware problem with the raid card? (we tests the individual drives, and they seem to pass the smart short & long tests). or can this be a firmware issue? maybe a driver issue on esx?, I just want to see some clarification why this is happening and makes it easier to explain to the supplier, why we think their server is defective
thanks!