L.root-servers.net, operated by ICANN, became the seventh of the root servers to have it’s IPv6 address records (AAAA) added into the DNS root-zone. The addition of IPv6 service is part of ICANN’s ongoing commitment to act as a leader in enabling IPv6 services throughout the DNS.

The new IPv6 address is 2001:500:3::42

To access this service DNS operators should update their hints files. In fact checking that you have the latest hints file on a regular basis is always good operational practice.

The latest hints files are available at the following URLs:

ftp://rs.internic.net/domain/named.root
ftp://ftp.internic.net/domain/named.root

The root servers currently answering on IPv6 are:

A.ROOT-SERVERS.NET 2001:503:ba3e::2:30
F.ROOT-SERVERS.NET 2001:500:2f::f
H.ROOT-SERVERS.NET 2001:500:1::803f:235
J.ROOT-SERVERS.NET 2001:503:c27::2:30
K.ROOT-SERVERS.NET 2001:7fd::1
L.ROOT-SERVERS.NET 2001:500:3::42
M.ROOT-SERVERS.NET 2001:dc3::35

At ICANN and for those who were aware of the situation this raised some eyebrows. Immediately after we turned off the old DNS server and noticed queries for the old address kept getting answers, we began looking into the issue by checking the DNS responses, reviewing routing system logs (including asking folks at Renesys and others for assistance), doing traceroutes from various points in the Internet, etc. Our investigation determined that the answers being provided by the machines now answering for the old L-root address were correct, returning proper referrals and answers including returning the correct address (199.7.83.42) for L-root, and as such, no immediate damage was being done. However, those answers should not have been returned period.

Why did the old L-root address keep answering?

As is well known, there are 13 root server IP addresses (and many DNS servers behind those addresses), referenced by a single letter (A through M) in the domain ROOT-SERVERS.NET. ICANN has operated L-root since 1997 or so. As part of an ongoing effort to improve L-root service, and after an extensive community consultation process, ICANN obtained a “critical infrastructure” block of addresses from ARIN and renumbered L-root out of a block that was originally intended for use at exchange points into the new block. Back in October 2007, ICANN announced this renumbering and continued to provide root name service on the old address for L-root while turning up a much beefier service on the new address. As part of the announced renumbering process, ICANN continued to answer DNS queries on the old address for 6 months, at which time root DNS service on 198.32.64.12 would cease.

On May 2, the 6 month clock expired, and we turned off root DNS service at the old address as planned. To our surprise, however, the registrant for the Exchange Point block, Bill Manning of “EP.NET, LLC.” (who under some relationship with the University of Southern California, Information Sciences Institute helps operate B.ROOT-SERVERS.NET) had entered into an agreement with a DNS hosting provider, CommunityDNS, to continue answering DNS root queries for the old L-root address.

As the folks at Renesys note, Bill Manning, as the current registrant for the superblock out of which for historical reasons the old L-root address was numbered, has the ability to re-provision the 198.32.64.0/24 address space as part of the Exchange Point block of address space. However, the continued availability of DNS root service on a “decommissioned” root server address violates the “Principle of Least Surprise”, at least for those who take a keen interest in the DNS infrastructure such as ICANN. Historically, when a root server has been renumbered, the root server operator maintains a DNS server at the old address for some period of time and then turns off service, with the address not being put back into use. The fact that queries to the old L-root address continued to be answered and ICANN, the operator of the L-root server, was unaware of agreements to continue service by a third party raises some significant areas of concern, not least of which is what should be done with IP addresses formerly used by root servers and who has the authority to make those sorts of decisions.

What did ICANN do?

After some discussion with the relevant parties, the DNS root service being provided at the old L-root address was discontinued. While undoubtedly a number of DNS queries were (and are) still being sent to the old L-root IP address, it is important to remember that the nature of the queries. When a typical DNS resolver first starts up, it uses information called the “root hints” as a list of known IP addresses for root servers. The DNS resolver will then ask one of those IP addresses (typically chosen at random) for an updated list, and then use that updated list from then on. This initial query is known as a “priming query” and only a fraction (1/13th to be precise) of priming queries from resolvers that had not updated their root hints file would have been affected — any queries following that priming query would not be affected since the answer to the priming query returns the correct root server address list.

ICANN has also been monitoring the results returned by these IP addresses through the entire time it was advertised, and believes it was always providing accurate root responses throughout its existence. ICANN continues to work with the root server operator community to improve monitoring and analysis of the root server system, aiming to ensure the continued security and stability of this critical component of the Internet.

So what does this mean for the Internet?

This incident highlights firstly how robust the DNS system is, that it continued to work flawlessly despite this issue. Given the existence of 13 root server IP addresses, only 1/13th of DNS resolvers that have not updated their root hints would have sent a priming query to the IP address formerly associated with the L-root server. However this incident also highlights how important it is for system administrators and DNS software developers to update their root hints as it becomes necessary. Thankfully, this usually only happens once every few years, and IANA sends announcements to the network operator communities when this information is updated.

ICANN, though its Root Server System Advisory Committee (RSSAC), is bringing some of the questions and issues raised by this incident to the root server community, who undoubtedly will want to consider this incident carefully.

http://dns.measurement-factory.com/tools/dsc/

“DSC” allows us to generate different views of the DNS queries we have been seeing at the L-ROOT systems. Both to the current IP address (199.7.83.42) and to the old address (198.32.64.12).

So let’s dispel these myths.

There are not 13 root servers.

What there are is there are many hundreds of root servers at over 130 physical locations in many different countries. There are twelve organisations responsible for the overall coordination of the management of these servers.

So where does the 13 number come from?

There is a technical design limitation that means thirteen is a practical maximum to the number of named authorities in the delegation data for the root zone. These named authorities are listed alphabetically, from a.root-servers.net through m.root-servers.net. Each has associated with it an IP address (and shortly some will have more than one as IPv6 is further rolled out).

But when we think of servers, we probably think of physical machines that sit on a desk, or perhaps lined up in racks in a specialised computing facility. By any measure, there are not 13 servers as there is not a correlation between the number of named authorities, and the number of servers.

The majority of named authorities are spread across multiple cities, often multiple countries. The “I” root, for example, is located in 25 different countries. But ignoring the physical diversity, even those authorities that are just in one physical location — the reality is they are comprised of networks of multiple servers that handle the millions of DNS queries the root servers receive every hour.

Another thing you may hear is that some of these root servers are just copies, whilst others are the “real” name servers. The reality is that every single root server is a copy, and none of them are more special than the others. In fact, the true master server from which the copies are made is not one of the public root servers.

So next time you hear there are 13 root servers, or that they are mostly in the US, just remember this map, courtesy of Patrik Fältström: