<div dir="ltr"><div><div>Personal experience. We had network issues here with things not routing or resolving correctly. We disabled IPV6 and all the issues went away. Not to say that is THE solution, it is more like a decent piece of duct-tape while finding a better solution. Since my company has not made the move to IPV6 and the rule is to disable IPV6 here. However, not everybody deploying devices attached to the network follow the rules. What we think is happening is that there is something here that is responding and replying to a lookup request on the IPV6 side and tossing that data back over the IPV4 stuff. It makes no logical sense, but it seems to be really prevalent on the Microsoft implementation of a network stack.<br>
<br></div>I asked about the implementation of DNS, because we do have issues with our Microsoft internal DNS server not being timely with its names table and we get legacy names where the devices have long since either disconnected or have a different IP. this is usually with DDNS (Windows Dynamic DNS) and WINS. Sadly my group is not allowed to actually manage or modify the internal DNS servers.<br>
<br></div>-Mic<br><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 27, 2014 at 12:05 PM, Matthew Campbell <span dir="ltr"><<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Good morning, still at it (day 3). I'm logging some of the work I'm doing so when you run across this you can refer back here for the process as it looks like there are a lot of parallel problems with DHCP/DNS that yield the same results.<br>
<br>With IPV6 disabled on the network interface I am still seeing the same problem.<br><br><b><span style="font-family:courier new,monospace">matt@Nala:220% nslookup leona<br>Server: <a href="http://senge.campbell.littlelionstudios.com" target="_blank">senge.campbell.littlelionstudios.com</a><br>
Address: 172.28.1.2<br><br>Name: <a href="http://leona.campbell.littlelionstudios.com" target="_blank">leona.campbell.littlelionstudios.com</a><br>Address: 172.28.3.151<br><br>matt@Nala:221% ping leona<br>Ping request could not find host leona. Please check the name and try again.<br>
<br>matt@Nala:222% <br></span></b><br>Option: Could it be a problem on my DNS server?<br><a href="http://stackoverflow.com/questions/330395/dns-problem-nslookup-works-ping-doesnt" target="_blank">http://stackoverflow.com/questions/330395/dns-problem-nslookup-works-ping-doesnt</a><br>
</div>Lots of possible answers here. I had already added the patch for the answer that looks the best (Windows adds .local to any hostname without a dot in it WTF?) by adding '.' to the domain resolution order on the Windows box so this is not it.<br>
<br></div><div>Option: "On Windows 2000 and later, if a request to your primary DNS server times
out, it switches to the secondary DNS server and stays with it for a
period of time. However, nslookup always connects to the primary."<br>As
the problem is intermittent and resolves after a period of time maybe
some of the boxes booted while the DNS server was offline and have
incorrect name servers? This case is not upheld by the continuing
intermittent failures.<br><br>Option: Could the computer be accessing another DNS server? I traced packets on my DNS server and it is not receiving the request from the host so something else is resolving the name. Supporting this option is the previous example where nslookup is returning the correct IP address and ssh / ping are getting a different answer. Flushing the dns cache on the host does not solve the problem so the incorrect answer is coming from somewhere external:<br>
<span style="font-family:arial,helvetica,sans-serif"><br></span><span style="font-family:courier new,monospace"><span style="font-family:arial,helvetica,sans-serif">No mention of aslan in the local cache:<br></span></span><b><span style="font-family:courier new,monospace">ipconfig /displaydns<br>
</span></b></div><div><b><span style="font-family:courier new,monospace"><br></span></b><div>Just to be safe dump the cache:<br></div><b><span style="font-family:courier new,monospace"></span></b><b><span style="font-family:courier new,monospace">ipconfig /flushdns<br>
</span></b></div><br></div>Still no joy, 2 different IP addresses are being returned.<br><div><div><b><span style="font-family:courier new,monospace">matt@Nala:214% ssh aslan<br><snip><br>The RSA host key for aslan has changed,<br>
and the key for the corresponding IP address 172.28.2.111<br>is unknown.<br><snip><br><br>matt@Nala:215% nslookup aslan<div><br>Server: <a href="http://senge.campbell.littlelionstudios.com" target="_blank">senge.campbell.littlelionstudios.com</a><br>
Address: 172.28.1.2<br><br>Name: <a href="http://aslan.campbell.littlelionstudios.com" target="_blank">aslan.campbell.littlelionstudios.com</a><br>Address: 172.28.2.100<br><br></div>matt@Nala:216% <br><br></span></b></div>
Option: there could be another DNS server on the network<br>
The host is set with the correct DNS lookup order. The primary DNS
server returns the correct hostname. The secondary and tertiary DNS
servers return 'host not found' so they are not providing the incorrect
DNS entry.<br><br>Option: there could be another service other than DNS providing the hostname lookup<br>This is supported by operations. nslookup can resolve nashost2, but Windows Explorer, ping and ssh can't find the server.<br>
<br>Yahoo!!! Found a clue in the /var/log/named.log file: It looks like when I coppied the dnssec keys over from my old server I didn't get the configuration quite right. But what's with the double dipping in the domain pool in that last line?<br>
<br><b><span style="font-family:courier new,monospace">Jan 27 04:39:34 senge named[2244]: 27-Jan-2014 04:39:34.217 dnssec: info: validating @0x7f3d904dd090: <a href="http://littlelionstudios.com" target="_blank">littlelionstudios.com</a> SOA: bad cache hit (<a href="http://littlelionstudios.com.dlv.isc.org/DLV" target="_blank">littlelionstudios.com.dlv.isc.org/DLV</a>)<br>
Jan 27 04:39:34 senge named[2244]: 27-Jan-2014 04:39:34.217 lame-servers: info: error (broken trust chain) resolving '<a href="http://nashost2.campbell.littlelionstudios.com.littlelionstudios.com/A/IN" target="_blank">nashost2.campbell.littlelionstudios.com.littlelionstudios.com/A/IN</a>': 208.109.255.44#53<br>
</span></b><br>Yahoo!!! Found another clue:<br></div><div>It turns out that Windows silently ignores applications it can run if your launcher is not of the right type. I was running inside a 32 bit shell and the NetBIOS tool is only a 64 bit app on Windows 8. Opening a command shell let me find the problem:<br>
<br><b><span style="font-family:courier new,monospace">C:\Users\matt>nbtstat -A 172.28.2.111<br>Ethernet:<br>Node IpAddress: [172.28.2.122] Scope Id: []<br> NetBIOS Remote Machine Name Table<br> Name Type Status<br>
---------------------------------------------<br> ASLAN <00> UNIQUE Registered<br> ASLAN <03> UNIQUE Registered<br> ASLAN <20> UNIQUE Registered<br>
CAMPBELL <00> GROUP Registered<br> CAMPBELL <1E> GROUP Registered<br> MAC Address = 00-00-00-00-00-00<br></span></b><br></div><div>Note that Aslan is mapped to IP address ...111 in NetBIOS. This is most likely the cause of the problems I have been seeing.<br>
</div><div><br>Now to install a NetBIOS server on the Linux host and link it to DNS; this should solve the problems. I'll post again in a couple of hours.<br><br>Matt<br><div><div><br></div></div></div></div><div class="gmail_extra">
<div>
<br clear="all"><div>---------<br><b style="color:rgb(51,102,102)">Matthew Campbell</b><br><font size="1">Storage Solution Consultant<br>Storage Design and Engineering<br></font><font face="Verdana" size="1"><br></font><b><span style="color:rgb(51,102,102)">Kaiser Permanente</span></b><br>
<font face="Verdana" size="1">IMG-Systems Integration</font><font size="1"><br>99 S. Oakland<br>Pasadena, CA 91101<br></font><br><font size="1"><a>626-564-7228</a> (office)<br></font><font face="Verdana" size="1"><a>8-338-7228</a> (tie-line)<br>
<a value="+18186918895">818-314-9897</a> (mobile phone)<br>Green Center 3-North, 031W29</font><font size="1"><br></font>---------<br><b><a style="color:rgb(51,102,102)" href="http://kp.org/thrive" target="_blank">kp.org/thrive</a></b><br>
</div>
<br><br></div><div><div><div class="gmail_quote">On Mon, Jan 27, 2014 at 8:39 AM, Matthew Campbell <span dir="ltr"><<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>Good suggestion Claude,<br><br></div><div>Unfortunately turning it back on made no difference. The issue is 'intermittent' which in this case means it's happening on every host until I fiddle with it then mysteriously gets better. It sometimes gets beter on its own in 20 minutes or so even without fiddling. It may have been coincidence that it started working in the same time span I was disabling IPV6.<br>
<br></div>To disable IPV6 you have to edit the registry. What I did was "Unbind IPv6 from a specific network adapter". Mic, was this a valid test?<br><a href="http://support.microsoft.com/kb/929852" target="_blank">http://support.microsoft.com/kb/929852</a><br>
<br>Matt<br><br></div><div class="gmail_extra"><div><br clear="all"><div>---------<br><b style="color:rgb(51,102,102)">Matthew Campbell</b><br><font size="1">Storage Solution Consultant<br>Storage Design and Engineering<br>
</font><font face="Verdana" size="1"><br>
</font><b><span style="color:rgb(51,102,102)">Kaiser Permanente</span></b><br><font face="Verdana" size="1">IMG-Systems Integration</font><font size="1"><br>99 S. Oakland<br>Pasadena, CA 91101<br></font><br><font size="1"><a>626-564-7228</a> (office)<br>
</font><font face="Verdana" size="1"><a>8-338-7228</a> (tie-line)<br><a value="+18186918895">818-314-9897</a> (mobile phone)<br>Green Center 3-North, 031W29</font><font size="1"><br></font>---------<br><b><a style="color:rgb(51,102,102)" href="http://kp.org/thrive" target="_blank">kp.org/thrive</a></b><br>
</div>
<br><br></div><div><div><div class="gmail_quote">On Mon, Jan 27, 2014 at 1:48 AM, Claude Felizardo <span dir="ltr"><<a href="mailto:cafelizardo@gmail.com" target="_blank">cafelizardo@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Matthew, just to be sure, have you tried turning IPv6 back on to<br>
confirm that the problem comes back? Anything else change? Reboot?<br>
<br>
I generally turn off IPv6 for now as I know some of my older equipment<br>
and applications do not support it.<br>
<br>
I just checked my router, I have IPv6 enabled but looks like I only<br>
have a four octet IP from my ISP so unless someone can give me a good<br>
reason to try IPv6 I'll probably just leave things as is and let<br>
others debug problems.<br>
<span><font color="#888888"><br>
Claude<br>
</font></span><div><div><br>
<br>
<br>
On Mon, Jan 27, 2014 at 1:32 AM, Matthew Campbell <<a href="mailto:dvdmatt@gmail.com" target="_blank">dvdmatt@gmail.com</a>> wrote:<br>
> The DNS server is bind on RHEL 6.4 with CENTOS 6.5 updates.<br>
><br>
> Turning off IPV6 suddenly made the bad times go away.<br>
><br>
> I have been fighting a ghost all weekend and thought I had things fixed a<br>
> number of times but they came back a couple of hours later. I'll take this<br>
> as a good sign but won't start celebrating yet.<br>
><br>
> Why did that work and what was going on? How did you know this was the<br>
> problem?<br>
><br>
> Was it intuition, prior experience or omniscience?<br>
><br>
> Matt<br>
><br>
><br>
> ---------<br>
> Matthew Campbell<br>
> Storage Solution Consultant<br>
> Storage Design and Engineering<br>
><br>
> Kaiser Permanente<br>
> IMG-Systems Integration<br>
> 99 S. Oakland<br>
> Pasadena, CA 91101<br>
><br>
> <a href="tel:626-564-7228" value="+16265647228" target="_blank">626-564-7228</a> (office)<br>
> 8-338-7228 (tie-line)<br>
> <a href="tel:818-314-9897" value="+18183149897" target="_blank">818-314-9897</a> (mobile phone)<br>
> Green Center 3-North, 031W29<br>
> ---------<br>
> <a href="http://kp.org/thrive" target="_blank">kp.org/thrive</a><br>
><br>
><br>
> On Sun, Jan 26, 2014 at 11:34 PM, Mic Chow <<a href="mailto:zen@netten.net" target="_blank">zen@netten.net</a>> wrote:<br>
>><br>
>> Is your network running IPV6; if not then disable it in the Windows box.<br>
>> What flavor is the DNS server; Microsoft?<br>
>><br>
>> Mic<br>
>><br>
>><br>
>> On 01/26/2014 10:50 PM, Matthew Campbell wrote:<br>
>><br>
>> Ok, I'm more than a little frustrated.<br>
>><br>
>> I installed a new DHCP/DNS server Friday. I washed all my Windows and<br>
>> can't do a thing with them:<br>
>><br>
>> matt@Nala:201% ipconfig /flushdns<br>
>> Windows IP Configuration<br>
>> Successfully flushed the DNS Resolver Cache.<br>
>><br>
>> matt@Nala:202% nslookup aslan<br>
>> Server: <a href="http://senge.campbell.littlelionstudios.com" target="_blank">senge.campbell.littlelionstudios.com</a><br>
>> Address: 172.28.1.2<br>
>> Name: <a href="http://aslan.campbell.littlelionstudios.com" target="_blank">aslan.campbell.littlelionstudios.com</a><br>
>> Address: 172.28.2.100<br>
>><br>
>> matt@Nala:203% ping aslan<br>
>> Pinging aslan [172.28.2.111] with 32 bytes of data:<br>
>> Reply from <a href="http://172.28.2.111" target="_blank">172.28.2.111</a>: bytes=32 time<1ms TTL=64<br>
>> Reply from <a href="http://172.28.2.111" target="_blank">172.28.2.111</a>: bytes=32 time<1ms TTL=64<br>
>><br>
>> matt@Nala:204%<br>
>><br>
>><br>
>> WTF?!? The IP address changed in the cache between the two calls or what?<br>
>> Of course things are not working. Any suggestions on how to diagnose?<br>
>><br>
>><br>
>> Matt<br>
>><br>
>> P.S. Here's another example:<br>
>><br>
>> matt@Nala:204% nslookup kiara<br>
>> Server: <a href="http://senge.campbell.littlelionstudios.com" target="_blank">senge.campbell.littlelionstudios.com</a><br>
>> Address: 172.28.1.2<br>
>><br>
>> Name: <a href="http://kiara.campbell.littlelionstudios.com" target="_blank">kiara.campbell.littlelionstudios.com</a><br>
>> Address: 172.28.2.102<br>
>><br>
>> matt@Nala:205% ssh kiara<br>
>> ssh: Could not resolve hostname kiara: hostname nor servname provided, or<br>
>> not known<br>
>><br>
>> matt@Nala:206%<br>
>><br>
>><br>
>> And here's the ipconfig:<br>
>><br>
>> matt@Nala:206% ipconfig /all<br>
>> Windows IP Configuration<br>
>> Host Name . . . . . . . . . . . . : Nala<br>
>> Primary Dns Suffix . . . . . . . :<br>
>> Node Type . . . . . . . . . . . . : Hybrid<br>
>> IP Routing Enabled. . . . . . . . : No<br>
>> WINS Proxy Enabled. . . . . . . . : No<br>
>> DNS Suffix Search List. . . . . . : <a href="http://campbell.littlelionstudios.com" target="_blank">campbell.littlelionstudios.com</a><br>
>> <a href="http://littlelionstudios.com" target="_blank">littlelionstudios.com</a><br>
>><br>
>> Ethernet adapter Ethernet:<br>
>> Connection-specific DNS Suffix . : <a href="http://campbell.littlelionstudios.com" target="_blank">campbell.littlelionstudios.com</a><br>
>> Description . . . . . . . . . . . : Broadcom NetLink (TM) Gigabit<br>
>> Ethernet<br>
>> Physical Address. . . . . . . . . : BC-5F-F4-4B-96-76<br>
>> DHCP Enabled. . . . . . . . . . . : Yes<br>
>> Autoconfiguration Enabled . . . . : Yes<br>
>> Link-local IPv6 Address . . . . . :<br>
>> fe80::6c04:e86b:ac8e:4d78%12(Preferred)<br>
>> IPv4 Address. . . . . . . . . . . : 172.28.2.122(Preferred)<br>
>> Subnet Mask . . . . . . . . . . . : 255.255.0.0<br>
>> Lease Obtained. . . . . . . . . . : Sunday, January 26, 2014 10:43:04<br>
>> PM<br>
>> Lease Expires . . . . . . . . . . : Monday, January 27, 2014 10:43:03<br>
>> PM<br>
>> Default Gateway . . . . . . . . . : 172.28.0.1<br>
>> DHCP Server . . . . . . . . . . . : 172.28.1.2<br>
>> DHCPv6 IAID . . . . . . . . . . . : 264003572<br>
>> DHCPv6 Client DUID. . . . . . . . :<br>
>> 00-01-00-01-18-96-BE-84-BC-5F-F4-4B-96-76<br>
>> DNS Servers . . . . . . . . . . . : 172.28.1.2<br>
>> 172.25.0.1<br>
>> 8.8.8.8<br>
>> Primary WINS Server . . . . . . . : 172.28.1.2<br>
>> NetBIOS over Tcpip. . . . . . . . : Enabled<br>
>><br>
>> Tunnel adapter <a href="http://isatap.campbell.littlelionstudios.com" target="_blank">isatap.campbell.littlelionstudios.com</a>:<br>
>> Media State . . . . . . . . . . . : Media disconnected<br>
>> Connection-specific DNS Suffix . : <a href="http://campbell.littlelionstudios.com" target="_blank">campbell.littlelionstudios.com</a><br>
>> Description . . . . . . . . . . . : Microsoft ISATAP Adapter<br>
>> Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0<br>
>> DHCP Enabled. . . . . . . . . . . : No<br>
>> Autoconfiguration Enabled . . . . : Yes<br>
>><br>
>> Tunnel adapter Local Area Connection* 11:<br>
>> Connection-specific DNS Suffix . :<br>
>> Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface<br>
>> Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0<br>
>> DHCP Enabled. . . . . . . . . . . : No<br>
>> Autoconfiguration Enabled . . . . : Yes<br>
>> IPv6 Address. . . . . . . . . . . :<br>
>> 2001:0:9d38:6ab8:cb6:220c:53e3:fd85(Preferred)<br>
>> Link-local IPv6 Address . . . . . :<br>
>> fe80::cb6:220c:53e3:fd85%14(Preferred)<br>
>> Default Gateway . . . . . . . . . : ::<br>
>> NetBIOS over Tcpip. . . . . . . . : Disabled<br>
>><br>
>> matt@Nala:205%<br>
>><br>
>> Thanks for any ideas on how to tackle this. The Linux upgrade including<br>
>> rebuilding 2 rack-mount servers from parts, OS installs, VM installs, App<br>
>> installs and configuration took 4 hours. I lost all day Saturday and all<br>
>> day Sunday to this d*** windows quirk.<br>
>><br>
>> Matt<br>
>><br>
>><br>
><br>
<br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>