I’ve had a dubious pleasure of seeing one of my clients lose the DNS resolution capability during my visit.
They have told me that they are using their ISPs (lightpath) DNS servers, which is OK, so long as it is up and when redundant resolvers are defined as public DNS servers such as IBM’s Quad 9, Cloudflare 1.1.1.1 or even Google’s 8.8.8.8
The problem was that when we’ve attempted to test nslookup using explicit server(s) with all of the above public DNS resolvers, they have all failed while the pings to the same IPs were succeeding.
It looked like this:
C:\ WINDOWS\system32>ping 9.9.9.9
Pinging 9.9.9.9 with 32 bytes of data:
Reply from 9.9.9.9: bytes=32 time=48ms TTL=51
Reply from 9.9.9.9: bytes=32 time=69ms TTL=51
Reply from 9.9.9.9: bytes=32 time=141ms TTL=51
Reply from 9.9.9.9: bytes=32 time=50ms TTL=51
Ping statistics for 9.9.9.9:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 48ms, Maximum = 141ms, Average = 77ms
C:\ WINDOWS\system32>
C:\WINDOWS\system32>nslookup
Default Server: mpnycdc-1.mps2000.com
Address: 10.0.0.217
> server 9.9.9.9
DNS request timed out.
timeout was 2 seconds.
Default Server: [9.9.9.9]
Address: 9.9.9.9
Server: [9.9.9.9]
Address: 9.9.9.9
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to [9.9.9.9] timed-out
> exit
C:\WINDOWS\system32>
I would like to suggest testing your ISPs for the DNS rerouting.
Please read https://labs.ripe.net/Members/babak_farrokhi/is-your-isp-hijacking-your-dns-traffic
And use a dnsdiag-1.6.4.win32.zip that could be found here https://github.com/farrokhi/dnsdiag/releases to test your DNS access.
You can run the commands from CMD using following syntax C:\path\dnsdiag-1.6.4.win32>dnstraceroute.exe -q -s #.#.#.# yahoo.com
where #.#.#.# is your public DNS server of choice.
My results look OK, if you receive responses NOT from the servers you are specifying in the command line, your DNS is being intercepted and rerouted.
C:\Users\UserName\Downloads\dnsdiag-1.6.4.win32\dnsdiag-1.6.4.win32>nslookup
Default Server: my.firewall
Address: 192.168.7.1
> server 9.9.9.9
Default Server: dns.quad9.net
Address: 9.9.9.9
Server: dns.quad9.net
Address: 9.9.9.9
Non-authoritative answer:
Name: www.google.com
Addresses: 2607:f8b0:4008:80d::2004
172.217.2.196
>
C:\Users\UserName\Downloads\dnsdiag-1.6.4.win32\dnsdiag-1.6.4.win32>dnstraceroute.exe -q -s 8.8.8.8 yahoo.com
1 *
2 *
3 *
4 *
5 *
6 *
7 *
8 *
9 *
10 *
11 google-public-dns-a.google.com (8.8.8.8) 76.179 ms
C:\Users\Username\Downloads\dnsdiag-1.6.4.win32\dnsdiag-1.6.4.win32>dnstraceroute.exe -q -s 1.1.1.1 yahoo.com
1 *
2 *
3 *
4 *
5 *
6 *
7 *
8 *
9 one.one.one.one (1.1.1.1) 22.131 ms
And while I may be fairly paranoid, this really is something I have seen: during this outage the maps on downdetector.com were listing major issues for large number of sites and ISPs, including L3, Lightpath etc… and when I have attempted to lookup same historical data today, these are not present.
The only few remnants are Sprint and Comcast:
Which is slightly unbelievable, but here we are…
Now, if I’d take the paranoia to a new level, I’d suggest that in the immediate aftermath of this event, the DNS intercepts will be paused and resumed few days later.
