Hi,
have here a 2 host, same configuration, one with a 10Gbit iSCSI attached Exomium box and one with a 10Gbit iSCSI QNAP TS-879 box (direct attached).
Have installed 6 Hitachi 2TB HDD in the QNAP. Did a lot of testing with QNAP in advance to move the box in production (FW:3.7.1).
Both installations ran fine and the performance was very well for a couple of month.
This is still true for the Exomium attached host, but after a firmware upgrade of the QNAP box (because a important fix came with this version),
the attached Vsphere 5.1 host freezed now 3 times in 3 weeks. The datastore disk access latency is bad, compared to the other box.
I ran "iostat -x 5" on the QNAP box, the HDD busy% is less than 30%, service time 3-5 ms, this seams normal.
Here are some log, no message on the Hardware console, according to the SNMP manager the Vsphere host stopped 11/23/12 21:02:05.
1.) Any idea to proof, if this performance behavior causes the frezze? Any way to do a better diagnose?
2.) Anybody else with Qnaps and Vsphere 5.1? Have ran "iostat", "top" etc., checked the logs, anything else to get more informations?
3.) Have still a free 10Gbit port on both sides, anybody with multipathing on Qnaps?
Thanks
Henri
2012-10-23T20:55:01Z crond[4569]: crond: USER root pid 901826 cmd /sbin/hostd-probe
2012-10-23T20:55:01Z syslog[901827]: starting hostd probing.
2012-10-23T20:55:16Z syslog[901827]: hostd probing is done.
2012-10-23T21:00:01Z crond[4569]: crond: USER root pid 902032 cmd /usr/lib/vmware/vmksummary/log-heartbeat.py
2012-10-23T21:00:01Z crond[4569]: crond: USER root pid 902033 cmd /sbin/hostd-probe
2012-10-23T21:00:01Z syslog[902035]: starting hostd probing.
2012-10-23T21:00:16Z syslog[902035]: hostd probing is done.
2012-10-23T21:01:01Z crond[4569]: crond: USER root pid 902117 cmd /sbin/auto-backup.sh
2012-10-24T05:57:12Z watchdog-vobd: [4514] Begin '/usr/lib/vmware/vob/bin/vobd ++min=0,max=100,group=uwdaemons', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000, bg_pid_file = ''
2012-10-24T05:57:12Z watchdog-vobd: Executing '/usr/lib/vmware/vob/bin/vobd ++min=0,max=100,group=host/vim/vmvisor/uwdaemons'
2012-10-24T05:57:14Z jumpstart: Using policy dir /etc/vmware/secpolicy
2012-10-24T05:57:14Z jumpstart: Parsed all objects
2012-10-24T05:57:14Z jumpstart: Objects defined and obsolete objects removed
2012-10-24T05:57:14Z jumpstart: Parsed all domain names
2012-10-23T21:01:40.070Z [2B09BB90 verbose 'vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information
2012-10-23T21:01:40.070Z [2B09BB90 verbose 'vim.PerformanceManager'] HostCtl Exception in stats collection. Turn on 'trivia' log for details
2012-10-23T21:02:00.026Z [2B05AB90 warning 'vim.PerformanceManager'] Calculated read I/O size 602112 for scsi0:0 is out of range -- 602112,prevBytes = 1577965419008 curBytes = 1577995524608 prevCommands = 56798322curCommands = 56798372
2012-10-23T21:02:00.063Z [2B05AB90 verbose 'vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information
2012-10-23T21:02:00.063Z [2B05AB90 verbose 'vim.PerformanceManager'] HostCtl Exception in stats collection. Turn on 'trivia' log for details
2012-10-23T21:02:03.585Z [2AD81B90 verbose 'SoapAdapter'] Responded to service state request
2012-10-23T21:02:04.477Z [2A981B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2012-10-23T21:02:19.680Z [2AD81B90 verbose 'DvsManager'] PersistAllDvsInfo called
2012-10-23T21:02:20.023Z [2B019B90 warning 'vim.PerformanceManager'] Calculated read I/O size 577308 for scsi0:0 is out of range -- 577308,prevBytes = 1577995524608 curBytes = 1578026699264 prevCommands = 56798372curCommands = 56798426
2012-10-23T21:02:20.057Z [2B019B90 verbose 'vim.PerformanceManager'] HostCtl Exception in stats collection: Sysinfo error on operation returned status : Not found. Please see the VMkernel log for detailed error information
2012-10-23T21:02:20.057Z [2B019B90 verbose 'vim.PerformanceManager'] HostCtl Exception in stats collection. Turn on 'trivia' log for details
2012-10-23T21:02:24.707Z [2B09BB90 info 'Snmpsvc'] UpdateStats: report cimom converter stats started
2012-10-23T21:02:24.707Z [2B09BB90 info 'Snmpsvc'] DumpStats: cimom stats file /tmp/.cvtcimsnmp.xml generated, size 298 bytes.
2012-10-23T21:02:24.707Z [2B09BB90 info 'Snmpsvc'] PublishReport: file /tmp/.cvtcimsnmp.xml published as /tmp/cvtcimsnmp.xml
2012-10-23T21:02:24.707Z [2B09BB90 info 'Snmpsvc'] DumpStats: cimom stats file published
2012-10-23T21:02:24.707Z [2B09BB90 info 'Snmpsvc'] NotifyAgent: write(51, /var/run/snmp.ctl, N) 1 bytes to snmpd
2012-10-23T21:02:24.707Z [2B09BB90 info 'Snmpsvc'] UpdateStats: report cimom converter stats completed
Section for VMware ESX, pid=5084, version=5.1.0, build=799733, option=Release
------ In-memory logs start --------
mem> 2012-10-24T06:00:56.807Z [FFEA8D20 verbose 'Default'] No update forwarding configured
mem> 2012-10-24T06:00:56.807Z [FFEA8D20 info 'Default'] Supported VMs 87
mem> 2012-10-24T06:00:56.807Z [FFEA8D20 info 'Handle checker'] Setting system limit of 2222
mem> 2012-10-24T06:00:56.807Z [FFEA8D20 info 'Handle checker'] Set system limit to 2222
mem> 2012-10-24T06:00:56.807Z [FFEA8D20 info 'Default'] Setting malloc mmap threshold to 32 k
mem> 2012-10-24T06:00:56.807Z [FFEA8D20 info 'Default'] getrlimit(RLIMIT_NPROC): curr=64 max=128, return code = Success
mem> 2012-10-24T06:00:56.807Z [FFEA8D20 info 'Default'] setrlimit(RLIMIT_NPROC): curr=128 max=128, return code = Success