<div dir="ltr"><div>Yes, but memtest86 came up clean so I don't think it's the RAM.<br><br></div>I have installed Phoronix which automates most of the tests I am aware of but it looks like it's a 2 week run. I set off the first of 15 tests this morning and it just finished...<br>
<br>Matt<br><br></div><div class="gmail_extra"><br clear="all"><div>---------<br><b style="color:rgb(51,102,102)">Matthew Campbell</b><br><font size="1">Storage Solution Consultant<br>Storage Design and Engineering<br></font><font face="Verdana" size="1"><br>
</font><b><span style="color:rgb(51,102,102)">Kaiser Permanente</span></b><br><font face="Verdana" size="1">IMG-Systems Integration</font><font size="1"><br>99 S. Oakland<br>Pasadena, CA 91101<br></font><br><font size="1"><a>626-564-7228</a> (office)<br>
</font><font face="Verdana" size="1"><a>8-338-7228</a> (tie-line)<br><a value="+18186918895">818-314-9897</a> (mobile phone)<br>Green Center 3-North, 031W29</font><font size="1"><br></font>---------<br><b><a style="color:rgb(51,102,102)" href="http://kp.org/thrive" target="_blank">kp.org/thrive</a></b><br>
</div>
<br><br><div class="gmail_quote">On Mon, Mar 3, 2014 at 1:08 PM, Jess Bermudes <span dir="ltr"><<a href="mailto:jbermudes@gmail.com" target="_blank">jbermudes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">For the RAM, would memtest86 be something like what you're looking for?<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 3, 2014 at 12:35 PM, Matthew Campbell <span dir="ltr"><<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Scott, that is an awesome diagram! It really points out the right tools to deep dive into each section of the system. Someone somewhere has already written a script to install, configure, run and interpret the output of the 30 tools. I don't want to re-invent the wheel ;)<br>
<br>Dan, swapping hardware is a good suggestion but I hope that running a benchmark tool should expose the problem component if there it may also cover driver issues, kernel issues, networking interactions, etc.. There are several closed source solutions along these lines. I was hoping someone on this list had experience with an open source product like the Phoronix suite<br>
<br><a href="http://www.phoronix-test-suite.com/" target="_blank">http://www.phoronix-test-suite.com/</a><br><a href="http://openbenchmarking.org/" target="_blank">http://openbenchmarking.org/</a><br><br></div><div>or any of the multitude of other open source benchmarks.<br>
<br><a href="http://en.wikipedia.org/wiki/Benchmark_%28computing%29#Common_benchmarks" target="_blank">http://en.wikipedia.org/wiki/Benchmark_%28computing%29#Common_benchmarks</a><br><br></div>Matt<br><br></div><div class="gmail_extra">
<div><br clear="all">
<div>---------<br><b style="color:rgb(51,102,102)">Matthew Campbell</b><br><font size="1">Storage Solution Consultant<br>Storage Design and Engineering<br></font><font face="Verdana" size="1"><br></font><b><span style="color:rgb(51,102,102)">Kaiser Permanente</span></b><br>
<font face="Verdana" size="1">IMG-Systems Integration</font><font size="1"><br>99 S. Oakland<br>Pasadena, CA 91101<br></font><br><font size="1"><a>626-564-7228</a> (office)<br></font><font face="Verdana" size="1"><a>8-338-7228</a> (tie-line)<br>
<a value="+18186918895">818-314-9897</a> (mobile phone)<br>Green Center 3-North, 031W29</font><font size="1"><br></font>---------<br><b><a style="color:rgb(51,102,102)" href="http://kp.org/thrive" target="_blank">kp.org/thrive</a></b><br>
</div>
<br><br></div><div><div><div class="gmail_quote">On Mon, Mar 3, 2014 at 8:49 AM, Dan Kegel <span dir="ltr"><<a href="mailto:dank@kegel.com" target="_blank">dank@kegel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I wonder if you could learn anything by swapping out the motherboard<br>
with a cheaper one.<br>
<div><div><br>
On Sun, Mar 2, 2014 at 7:31 PM, Matthew Campbell <<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>> wrote:<br>
> Yep. Tried that with the RAM but the Mobo and CPU are the latest and I<br>
> don't want to blow another grand on duplicates...<br>
><br>
> Matt<br>
><br>
> ---------<br>
> Matthew Campbell<br>
> Storage Solution Consultant<br>
> Storage Design and Engineering<br>
><br>
> Kaiser Permanente<br>
> IMG-Systems Integration<br>
> 99 S. Oakland<br>
> Pasadena, CA 91101<br>
><br>
> <a href="tel:626-564-7228" value="+16265647228" target="_blank">626-564-7228</a> (office)<br>
> 8-338-7228 (tie-line)<br>
> <a href="tel:818-314-9897" value="+18183149897" target="_blank">818-314-9897</a> (mobile phone)<br>
> Green Center 3-North, 031W29<br>
> ---------<br>
> <a href="http://kp.org/thrive" target="_blank">kp.org/thrive</a><br>
><br>
><br>
> On Sun, Mar 2, 2014 at 5:01 PM, Dan Kegel <<a href="mailto:dank@kegel.com" target="_blank">dank@kegel.com</a>> wrote:<br>
>><br>
>> Swapping out part by part until the problem goes away might be your best<br>
>> bet.<br>
>><br>
>> Am 02.03.2014 15:24 schrieb "Matthew Campbell" <<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>>:<br>
>><br>
>>> Does anyone have a hardware diagnostic tool they like, preferably open<br>
>>> source? I have been fighting a host for two weeks now and after finding and<br>
>>> submitted 2 kernel bugs have begun to suspect that the problems I am running<br>
>>> into are being exposed by a hardware failure.<br>
>>><br>
>>> The system appears to be running fine, but every 10-15 seconds will zone<br>
>>> out for a couple of seconds. At first I thought it was a BTRFS bug, and the<br>
>>> errors I was seeing turned out to be just that.<br>
>>><br>
>>> Once they were fixed the freezing kept on. Further poking uncovered a<br>
>>> NFS bug in its interaction with the underlying filesystem, but having also<br>
>>> patched the kernel for that the poor performance continues.<br>
>>><br>
>>> Now I'm starting to see errors of this sort in my syslog:<br>
>>><br>
>>> 2014-03-02T22:39:00.262Z cpu6:34527)WARNING: LinScsi:<br>
>>> SCSILinuxQueueCommand:1207: queuecommand failed with status = 0x1056 Unknown<br>
>>> status vmhba0:0:0:0 (driver name: ahci) - Message repeated 4 times<br>
>>> 2014-03-02T22:39:00.262Z cpu2:32791)ScsiDeviceIO: 2324:<br>
>>> Cmd(0x412e8088eac0) 0x4d, CmdSN 0x784 from world 0 to dev<br>
>>> "t10.ATA_____INTEL_SSDSC2BW240A4_____________________CVDA341000752403GN__"<br>
>>> failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.<br>
>>> 2014-03-02T22:39:00.275Z cpu2:32784)ScsiDeviceIO: 2324:<br>
>>> Cmd(0x412e80842b00) 0x28, CmdSN 0x51c3 from world 32878 to dev<br>
>>> "t10.ATA_____INTEL_SSDSC2BW240A4_____________________CVDA341000752403GN__"<br>
>>> failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.<br>
>>><br>
>>> Could my SSD be failing? But I just replaced the previous boot disk as<br>
>>> it looked like it was failing...<br>
>>><br>
>>> Device sense code D:0x8 equates to 08h BUSY according to these docs:<br>
>>><br>
>>> <a href="http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=289902" target="_blank">http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=289902</a><br>
>>><br>
>>> It could be a MOBO issue with the SATA port or even the CPU or RAM. Ugh.<br>
>>><br>
>>> I tried memtest86 and all passed...<br>
>>><br>
>>> Any suggestions on a full-system hardware test suite would be much<br>
>>> appreciated.<br>
>>><br>
>>> Matt<br>
>>><br>
><br>
<br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>