cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
vt100
Community Champion

DNS proxying or rerouting by ISPs

I’ve had a dubious pleasure of seeing one of my clients lose the DNS resolution capability during my visit.

They have told me that they are using their ISPs (lightpath) DNS servers, which is OK, so long as it is up and when redundant resolvers are defined as public DNS servers such as IBM’s Quad 9, Cloudflare 1.1.1.1 or even Google’s 8.8.8.8

 

The problem was that when we’ve attempted to test nslookup using explicit server(s) with all of the above public DNS resolvers, they have all failed while the pings to the same IPs were succeeding.

 

It looked like this:

 

C:\ WINDOWS\system32>ping 9.9.9.9

 

Pinging 9.9.9.9 with 32 bytes of data:

Reply from 9.9.9.9: bytes=32 time=48ms TTL=51

Reply from 9.9.9.9: bytes=32 time=69ms TTL=51

Reply from 9.9.9.9: bytes=32 time=141ms TTL=51

Reply from 9.9.9.9: bytes=32 time=50ms TTL=51

 

Ping statistics for 9.9.9.9:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 48ms, Maximum = 141ms, Average = 77ms

 

C:\ WINDOWS\system32>

 

C:\WINDOWS\system32>nslookup

Default Server:  mpnycdc-1.mps2000.com

Address:  10.0.0.217

 

> server 9.9.9.9

DNS request timed out.

    timeout was 2 seconds.

Default Server:  [9.9.9.9]

Address:  9.9.9.9

 

> www.google.com

Server:  [9.9.9.9]

Address:  9.9.9.9

 

DNS request timed out.

    timeout was 2 seconds.

DNS request timed out.

    timeout was 2 seconds.

DNS request timed out.

    timeout was 2 seconds.

DNS request timed out.

    timeout was 2 seconds.

*** Request to [9.9.9.9] timed-out

> exit

 

C:\WINDOWS\system32>

 

 

I would like to suggest testing your ISPs for the DNS rerouting.

 

Please read https://labs.ripe.net/Members/babak_farrokhi/is-your-isp-hijacking-your-dns-traffic

 

And use a  dnsdiag-1.6.4.win32.zip that could be found here https://github.com/farrokhi/dnsdiag/releases to test your DNS access.

 

You can run the commands from CMD using following syntax C:\path\dnsdiag-1.6.4.win32>dnstraceroute.exe -q -s #.#.#.# yahoo.com

where #.#.#.# is your public DNS server of choice.

 

My results look OK, if you receive responses NOT from the servers you are specifying in the command line, your DNS is being intercepted and rerouted.

 

C:\Users\UserName\Downloads\dnsdiag-1.6.4.win32\dnsdiag-1.6.4.win32>nslookup

Default Server:  my.firewall

Address:  192.168.7.1

 

> server 9.9.9.9

Default Server:  dns.quad9.net

Address:  9.9.9.9

 

> www.google.com

Server:  dns.quad9.net

Address:  9.9.9.9

 

Non-authoritative answer:

Name:    www.google.com

Addresses:  2607:f8b0:4008:80d::2004

          172.217.2.196

 

 

C:\Users\UserName\Downloads\dnsdiag-1.6.4.win32\dnsdiag-1.6.4.win32>dnstraceroute.exe -q -s 8.8.8.8 yahoo.com

1        *

2        *

3        *

4        *

5        *

6        *

7        *

8        *

9        *

10       *

11      google-public-dns-a.google.com (8.8.8.8) 76.179 ms

 

C:\Users\Username\Downloads\dnsdiag-1.6.4.win32\dnsdiag-1.6.4.win32>dnstraceroute.exe -q -s 1.1.1.1 yahoo.com

1        *

2        *

3        *

4        *

5        *

6        *

7        *

8        *

9       one.one.one.one (1.1.1.1) 22.131 ms

 

And while I may be fairly paranoid, this really is something I have seen: during this outage the maps on downdetector.com were listing major issues for large number of sites and ISPs, including L3, Lightpath etc… and when I have attempted to lookup same historical data today, these are not present.

The only few remnants are Sprint and Comcast:Sprint_outage.jpg

 

comcast_outage.jpg

 

Which is slightly unbelievable, but here we are…

Now, if I’d take the paranoia to a new level, I’d suggest that in the immediate aftermath of this event, the DNS intercepts will be paused and resumed few days later.

 

2 Replies
AlecTrevelyan
Community Champion


@vt100 wrote:

I’ve had a dubious pleasure of seeing one of my clients lose the DNS resolution capability during my visit.

That's one of every Consultant's pet hates! You attend a client site and something happens completely unrelated to what you're working on, but you inevitably get asked if it's something to do with you and then get dragged into helping out / proving your innocence!

 


@vt100 wrote:

They have told me that they are using their ISPs (lightpath) DNS servers, which is OK, so long as it is up and when redundant resolvers are defined as public DNS servers such as IBM’s Quad 9, Cloudflare 1.1.1.1 or even Google’s 8.8.8.8

 

The problem was that when we’ve attempted to test nslookup using explicit server(s) with all of the above public DNS resolvers, they have all failed while the pings to the same IPs were succeeding.

 

It looked like this:

 

C:\ WINDOWS\system32>ping 9.9.9.9

 

Pinging 9.9.9.9 with 32 bytes of data:

Reply from 9.9.9.9: bytes=32 time=48ms TTL=51

Reply from 9.9.9.9: bytes=32 time=69ms TTL=51

Reply from 9.9.9.9: bytes=32 time=141ms TTL=51

Reply from 9.9.9.9: bytes=32 time=50ms TTL=51

 

Ping statistics for 9.9.9.9:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 48ms, Maximum = 141ms, Average = 77ms

 

C:\ WINDOWS\system32>

 

C:\WINDOWS\system32>nslookup

Default Server:  mpnycdc-1.mps2000.com

Address:  10.0.0.217

 

> server 9.9.9.9

DNS request timed out.

    timeout was 2 seconds.

Default Server:  [9.9.9.9]

Address:  9.9.9.9

 

> www.google.com

Server:  [9.9.9.9]

Address:  9.9.9.9

 

DNS request timed out.

    timeout was 2 seconds.

DNS request timed out.

    timeout was 2 seconds.

DNS request timed out.

    timeout was 2 seconds.

DNS request timed out.

    timeout was 2 seconds.

*** Request to [9.9.9.9] timed-out

> exit

 

C:\WINDOWS\system32>

Sorry if this is a basic question, but are the systems from which you were testing DNS allowed to send the required traffic through the client's firewall(s) to these specific external DNS servers?

 

I generally wouldn't expect firewall policy to allow desktop machines and non DNS servers to be able to resolve DNS externally. While internal DNS servers might be restricted to the ISP's DNS servers and maybe a few other public servers (which are possibly different to the ones you're testing).

 

vt100
Community Champion

Yeah, in this case, DNS lookup as well as ICMP from any machines inside that network segment was permitted.

This being a guest network that was sharing their ISP.

 

I too expect to see the production outbound traffic only from forwarders. In the absence of dedicated DNS infrastructure and common enough to see are the domain controllers with DNS services acting in this capacity.

 

Unfortunately, at the time it did happen, I did not have my production laptop with me, where I am using DNSCrypt local proxy with forwarding. It would've been pretty conclusive if I would've been able to resolve from it.