<div dir="ltr"><div><div><div><div>Hey Claude, thanks for your thoughts.<br><br></div>I have had bad (really bad) luck with OCZ so current I only do Intel SSDs. I have not had one fail yet.<br><br>As I replaced a Seagate HD with the SSD and am getting the same errors it makes me think this is a controller issue. I had the boot drive running off the RAID port on the motherboard (a Supermicro X9SAE-V), then moved it to a non-RAID Mobo port with no change in symptoms.<br>
</div><br></div>This is the only drive on the system. I have moved all compute into 2 redundant nodes and storage on a separate set of servers.<br><br></div>I am looking for a tool to first confirm that the server is freezing up, then to determine what component is failing<br>
<br>There's nothing I have seen in the hardware logs or OS logs other than the disk errors.<br><br>Matt<br><br></div><div class="gmail_extra"><br clear="all"><div>---------<br><b style="color:rgb(51,102,102)">Matthew Campbell</b><br>
<font size="1">Storage Solution Consultant<br>Storage Design and Engineering<br></font><font face="Verdana" size="1"><br></font><b><span style="color:rgb(51,102,102)">Kaiser Permanente</span></b><br><font face="Verdana" size="1">IMG-Systems Integration</font><font size="1"><br>
99 S. Oakland<br>Pasadena, CA 91101<br></font><br><font size="1"><a>626-564-7228</a> (office)<br></font><font face="Verdana" size="1"><a>8-338-7228</a> (tie-line)<br><a value="+18186918895">818-314-9897</a> (mobile phone)<br>
Green Center 3-North, 031W29</font><font size="1"><br></font>---------<br><b><a style="color:rgb(51,102,102)" href="http://kp.org/thrive" target="_blank">kp.org/thrive</a></b><br></div>
<br><br><div class="gmail_quote">On Mon, Mar 3, 2014 at 12:17 AM, Claude Felizardo <span dir="ltr"><<a href="mailto:cafelizardo@gmail.com" target="_blank">cafelizardo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">can you try the drives in another computer just to rule them out? SATA on MB or card?<div><br></div><div>you sure there isn’t a bug with the firmware on the drives? I had a problem with drives in a RAID - bug would show up ever few weeks or maybe a month when it was trying to do calibration while in RAID config and it would knock the drive offline. work around was to reboot before that period until i stumbled on posts saying firmware was bad and an updated fixed it.</div>
<div><br></div><div>Oh wait, all SSD? What brand?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Claude</div></font></span><div><div class="h5"><div><br></div><div><br></div><div><br><div><div>On Mar 2, 2014, at 7:31 PM, Matthew Campbell <<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr"><div>Yep. Tried that with the RAM but the Mobo and CPU are the latest and I don't want to blow another grand on duplicates...<br><br></div>Matt<br></div><div class="gmail_extra">
<br clear="all"><div>---------<br>
<b style="color:rgb(51,102,102)">Matthew Campbell</b><br><font size="1">Storage Solution Consultant<br>Storage Design and Engineering<br></font><font face="Verdana" size="1"><br></font><b><span style="color:rgb(51,102,102)">Kaiser Permanente</span></b><br>
<font face="Verdana" size="1">IMG-Systems Integration</font><font size="1"><br>99 S. Oakland<br>Pasadena, CA 91101<br></font><br><font size="1"><a>626-564-7228</a> (office)<br></font><font face="Verdana" size="1"><a>8-338-7228</a> (tie-line)<br>
<a value="+18186918895">818-314-9897</a> (mobile phone)<br>Green Center 3-North, 031W29</font><font size="1"><br></font>---------<br><b><a style="color:rgb(51,102,102)" href="http://kp.org/thrive" target="_blank">kp.org/thrive</a></b><br>
</div>
<br><br><div class="gmail_quote">On Sun, Mar 2, 2014 at 5:01 PM, Dan Kegel <span dir="ltr"><<a href="mailto:dank@kegel.com" target="_blank">dank@kegel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">Swapping out part by part until the problem goes away might be your best bet.<br>
</p>
<div class="gmail_quote">Am 02.03.2014 15:24 schrieb "Matthew Campbell" <<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>>:<div><div><br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div><div><div><div>Does anyone have a hardware diagnostic tool they like, preferably open source? I have been fighting a host for two weeks now and after finding and submitted 2 kernel bugs have begun to suspect that the problems I am running into are being exposed by a hardware failure.<br>
<br></div>The system appears to be running fine, but every 10-15 seconds will zone out for a couple of seconds. At first I thought it was a BTRFS bug, and the errors I was seeing turned out to be just that.<br><br>Once they were fixed the freezing kept on. Further poking uncovered a NFS bug in its interaction with the underlying filesystem, but having also patched the kernel for that the poor performance continues.<br>
<br></div>Now I'm starting to see errors of this sort in my syslog:<br><br>2014-03-02T22:39:00.262Z cpu6:34527)WARNING: LinScsi: SCSILinuxQueueCommand:1207: queuecommand failed with status = 0x1056 Unknown status vmhba0:0:0:0 (driver name: ahci) - Message repeated 4 times<br>
2014-03-02T22:39:00.262Z cpu2:32791)ScsiDeviceIO: 2324: Cmd(0x412e8088eac0) 0x4d, CmdSN 0x784 from world 0 to dev "t10.ATA_____INTEL_SSDSC2BW240A4_____________________CVDA341000752403GN__" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.<br>
2014-03-02T22:39:00.275Z cpu2:32784)ScsiDeviceIO: 2324: Cmd(0x412e80842b00) 0x28, CmdSN 0x51c3 from world 32878 to dev "t10.ATA_____INTEL_SSDSC2BW240A4_____________________CVDA341000752403GN__" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.<br>
<br></div>Could my SSD be failing? But I just replaced the previous boot disk as it looked like it was failing...<br><br></div><div>Device sense code D:0x8 equates to <span><code>08h BUSY according to these docs:<br>
<a href="http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=289902" target="_blank">http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=289902</a><br>
<br></code></span></div>It could be a MOBO issue with the SATA port or even the CPU or RAM. Ugh.<br><br>I tried memtest86 and all passed...<br><br></div>Any suggestions on a full-system hardware test suite would be much appreciated.<br>
<br>Matt<br><br></div>
</blockquote></div></div></div>
</blockquote></div><br></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br></div>