Ghosts of Root Servers Past

by David Conrad on May 19, 2008

As noticed by some in the Internet network operations community, at the beginning of May an odd event occurred as ICANN ended DNS service on the IP address formerly associated with L.ROOT-SERVERS.NET (“L-root”). Specifically, as ICANN turned off the DNS service at the address formerly used by the L-root, 198.32.64.12 (and the routing announcement by ICANN for 198.32.64.0/24), DNS root queries sent to that address instead of the new L-root address (199.7.83.42) continued to be answered.

At ICANN and for those who were aware of the situation this raised some eyebrows. Immediately after we turned off the old DNS server and noticed queries for the old address kept getting answers, we began looking into the issue by checking the DNS responses, reviewing routing system logs (including asking folks at Renesys and others for assistance), doing traceroutes from various points in the Internet, etc. Our investigation determined that the answers being provided by the machines now answering for the old L-root address were correct, returning proper referrals and answers including returning the correct address (199.7.83.42) for L-root, and as such, no immediate damage was being done. However, those answers should not have been returned period.

Why did the old L-root address keep answering?

As is well known, there are 13 root server IP addresses (and many DNS servers behind those addresses), referenced by a single letter (A through M) in the domain ROOT-SERVERS.NET. ICANN has operated L-root since 1997 or so. As part of an ongoing effort to improve L-root service, and after an extensive community consultation process, ICANN obtained a “critical infrastructure” block of addresses from ARIN and renumbered L-root out of a block that was originally intended for use at exchange points into the new block. Back in October 2007, ICANN announced this renumbering and continued to provide root name service on the old address for L-root while turning up a much beefier service on the new address. As part of the announced renumbering process, ICANN continued to answer DNS queries on the old address for 6 months, at which time root DNS service on 198.32.64.12 would cease.

On May 2, the 6 month clock expired, and we turned off root DNS service at the old address as planned. To our surprise, however, the registrant for the Exchange Point block, Bill Manning of “EP.NET, LLC.” (who under some relationship with the University of Southern California, Information Sciences Institute helps operate B.ROOT-SERVERS.NET) had entered into an agreement with a DNS hosting provider, CommunityDNS, to continue answering DNS root queries for the old L-root address.

As the folks at Renesys note, Bill Manning, as the current registrant for the superblock out of which for historical reasons the old L-root address was numbered, has the ability to re-provision the 198.32.64.0/24 address space as part of the Exchange Point block of address space. However, the continued availability of DNS root service on a “decommissioned” root server address violates the “Principle of Least Surprise”, at least for those who take a keen interest in the DNS infrastructure such as ICANN. Historically, when a root server has been renumbered, the root server operator maintains a DNS server at the old address for some period of time and then turns off service, with the address not being put back into use. The fact that queries to the old L-root address continued to be answered and ICANN, the operator of the L-root server, was unaware of agreements to continue service by a third party raises some significant areas of concern, not least of which is what should be done with IP addresses formerly used by root servers and who has the authority to make those sorts of decisions.

What did ICANN do?

After some discussion with the relevant parties, the DNS root service being provided at the old L-root address was discontinued. While undoubtedly a number of DNS queries were (and are) still being sent to the old L-root IP address, it is important to remember that the nature of the queries. When a typical DNS resolver first starts up, it uses information called the “root hints” as a list of known IP addresses for root servers. The DNS resolver will then ask one of those IP addresses (typically chosen at random) for an updated list, and then use that updated list from then on. This initial query is known as a “priming query” and only a fraction (1/13th to be precise) of priming queries from resolvers that had not updated their root hints file would have been affected — any queries following that priming query would not be affected since the answer to the priming query returns the correct root server address list.

ICANN has also been monitoring the results returned by these IP addresses through the entire time it was advertised, and believes it was always providing accurate root responses throughout its existence. ICANN continues to work with the root server operator community to improve monitoring and analysis of the root server system, aiming to ensure the continued security and stability of this critical component of the Internet.

So what does this mean for the Internet?

This incident highlights firstly how robust the DNS system is, that it continued to work flawlessly despite this issue. Given the existence of 13 root server IP addresses, only 1/13th of DNS resolvers that have not updated their root hints would have sent a priming query to the IP address formerly associated with the L-root server. However this incident also highlights how important it is for system administrators and DNS software developers to update their root hints as it becomes necessary. Thankfully, this usually only happens once every few years, and IANA sends announcements to the network operator communities when this information is updated.

ICANN, though its Root Server System Advisory Committee (RSSAC), is bringing some of the questions and issues raised by this incident to the root server community, who undoubtedly will want to consider this incident carefully.

{ 15 comments }

bill manning 05.19.08 at 8:28 pm

there was agreement between root operators to collect data on some retired addresses for DITL-2008. the reasons for collecting that data was to check for correlation on querying nodes. The problems have been identified in NANOG presentations and at WIDE/CAIDA workshops over the
years… http://www.caida.org/workshops/wide/0611/ has one of my contributions there.
It is likely that an analysis of the 2008 data will be published later this year.

I understand that ISC is running a data collection service (SIE) and has asked for permission to
collect this type of data on an ongoing basis.

A little unorthodox? yes. A surprise, hardly.

David Conrad 05.20.08 at 6:11 am

Bill,

Speaking personally:

First of all, it isn’t clear to me why anyone would believe a core component of the Internet infrastructure, even one that has been decommissioned, would be considered a personal playtoy that could be studied and analyzed without open, transparent, and accountable mechanisms to ensure that infrastructure wasn’t being abused for malicious or self-serving purposes. If you look at the discussion on (say) Slashdot, you’ll note the vast majority of commenters assume the purpose of this event was nefarious in some way. This might suggest there is sensitivity and a need for openness, transparency, and accountability.

Like, oh say, this.

Secondly, even if one were to assume it appropriate to experiment with a retired root server address, data collection does not mean responding to queries, does notmean privately negotiating with a third party to respond to those queries, does not mean becoming a distribution master for that third party, etc.

Finally, the fact that you did this without ICANN, the L-root operator’s, awareness or agreement, would tend to imply that it was indeed a surprise.

Regards,
-drc

bill manning 05.24.08 at 8:57 pm

neferious indeed. can you explain why ICANN was using EP.NET space to begin with instead of renumbering promptly, like K, and M did?

wrt ICANN not knowing, either Mr. Crain is not talking to you, is not reading his email, or there are other problems. He was notified that we would be standing up services here, at least as long as the DITL collection was occuring. if you’d like to see the email, and he no longer has a copy, you should be able to get one from Matt Larson, KC, or even me.

If you believe that CAIDA and their projects are “personal playtoys” – that is your perogative. if you believe that EP.NET can not use its address space, for what ever reason, then I’d really like to have that discussion. If you think that EP.NET should not use its address space, that would be a useful discussion as well.

John Crain 05.25.08 at 5:22 pm

Bill,

It was not my understanding at all that ENET would stand up anything related to L-root. My understanding was that EPNET would be announcing and listening to queries at the old M address and that ISI would be doing the same for B.

As the operator of L-ROOT and members of OARC we took part in the CAIDA project and supplied data streams from the new and old IP address as we were still answering queries at that time. Something of which you were fully aware.

I am willing to believe that there was some misunderstanding of EPNETS role in that project as it relates to L-ROOT. However that does not explain the recent situation. That project required us to deliver streams of data to CAIDA/OARC for a 48 hour period only. A period of time which is long past

For those reading this blog I do encourage you to follow CAIDAs work at http://www.caida.org/ and specifically the DITL project.

David Conrad 05.27.08 at 7:30 am

an you explain why ICANN was using EP.NET space to begin with instead of renumbering promptly, like K, and M did?

My understanding was that the 192.32.0.0/16 block was allocated to USC ISI long ago for the NSF-sponsored “NAPs” and subsequently used more generally for Exchange Points. As an expediency, and presumably because it was assumed root servers would be located at exchange points, when the last additional root servers were allocated, they were allocated address space from that USC ISI block. Since the L-root server remained physically located at USC ISI (as opposed to the other root servers you mention), it isn’t clear to me why renumbering it out of that block would have been time critical. After other root servers that were in the queue ahead of L-root were renumbered and when L-root was physically (and network topologically) moved, it was renumbered.

As to why and how “EP.NET, LLC.” came into possession of 192.32.0.0/16, that is probably a topic best explored elsewhere.

Tony Toews 05.28.08 at 5:34 pm

Please use black on white letters for these pages. Not grey on white.

kc 05.28.08 at 11:26 pm

maybe i’m naive, but i’m extremely surprised to see CAIDA and DITL 2008 implicated in Bill’s justification, not only because we never received any DITL2008 data from Bill, but also because we knew nothing of any plans to impersonate root dns servers. Bill refers to our 2006 workshop where he described his honeypot at the old b-root address, which responded with ICMP port unreachables. but at this year’s DITL2008 planning workshop, i only remember Bill and John Crain discussing collecting data from old roots, not setting up responders. we also had a pre-DITL prep conference call on 17 march 2008, which Bill missed but sent mail saying:

Date: Mon, 17 Mar 2008 15:26:21 +0000
From: bmanning@vacation.karoshi.com
To: Keith Mitchell
Cc: mlarson@verisign.com, john.crain@icann.org, kato@wide.ad.jp, kc@caida.org
Subject: Re: DITL08 Pre-collection conference call

not going to make the call. have set up and there is active collection going
for the old “M” and “L” nodes – working on the old “B” node. data looks like
~ 200M/hr – will store locally and figure out details later. Hope that John
and Matt have similar collections running for their old numbers as well.

–bill

if by “active collection” (as opposed to ‘passive collection’, the canonical description for tcpdump/dnscap), he meant to notify icann and verisign and caida that he was setting up a responder on old-L, i’m not sure how we were supposed to figure that out. it would have never occurred to me, even having heard the honeypot talk in 2006 (which used tcpdump and icmp port unreachables). that’s the only email i have on the topic, so if Bill is referring to some other notification, i hope he posts it. (did he have active responders on old-M and old-B?)

we’d be delighted to have Bill contribute 2008 data to DITL as he implied he would. but the connection to DNS root impersonation both surprises and troubles me. of course it’s even more troubling that we haven’t figured out stewardship of old root IP addresses yet, but since we haven’t really figured out stewardship of any IP addresses yet, i can’t claim surprise.

kc 05.29.08 at 4:45 pm

regarding the call for “open, transparent, accountable” discussion of
this topic, next week’s OARC meeting (open to all, at no charge) kicks off with a Renesys talk on
“Who is Manning the L-Root.” given usual OARC meeting attendance,
it should be quite informative.

AT 05.30.08 at 1:31 pm

Facts like this expose to the sun how ICANN has under control what it is doing.
It is time to close this corporation and pass the rights to manage TLDs to various confederated organisms distributed at least by continent when not by nation.
It makes really no sense to have a single corp under the supervision of a single nation.

The specific case demonstrates not the strength, au-contraire!, the fragility of the actual architecture,

Hope Mr. President doesn’t decide know to democratically switch off ccTLD for the country I live because of my comment. ;-)

David Conrad 05.30.08 at 3:00 pm

Facts like this expose to the sun how ICANN has under control what it is doing.

Actually, facts like these expose how individuals associated with the root servers are independent actors and how the decentralized nature of the Internet routing system currently works.

It makes really no sense to have a single corp under the supervision of a single nation.

Fortunately, ICANN is “under the supervision” of a wide array of stakeholders and constituencies.

Hope Mr. President doesn’t decide know to democratically switch off ccTLD for the country I live because of my comment.

This comment indicates a fundamental misunderstanding of how changes to the root of the DNS are made as well as the relationships between ICANN, the ccTLDs, the root server operators, and (presumably) the U.S. government. Ignoring for the moment the reality that any change as you describe, even if it got that far (which it wouldn’t) would have to be approved by ICANN’s multi-national and multi-cultural board, the fact that the root servers are independently operated is usually identified as a reason why “Mr. President” (whoever that might be) can’t decide (democratically or not) to “switch off” a ccTLD and have that change propagated to the Internet as a whole.

bill manning 06.03.08 at 8:03 am

David,
You are mistaken. 198.32.0.0/16 was never assigned to USC/ISI. It came with me when Jon Postel asked me to work for him and he agreed to its use for exchanges, root servers, and other network infrastructure.

bill manning 06.03.08 at 8:05 am

KC,
I’d be more than happy to send you the data, have not been able to get past the OARC authentication thresholds.

tom 06.26.08 at 5:13 pm

[…] former peeps over at Yahoo just released 10 more components, 3 Flash and 5 Flex components. The also fixed some of the bugs […]

della 06.26.08 at 11:58 pm

jyxpearl 07.20.08 at 8:45 pm

Manufacturer,wholesaler of Pearl jewelry,freshwater pearl beads,akoya pearls,pearl necklaces,
wedding & bridal jewelry,crystal necklace,wholesale beads,fine jewelry.

http://www.jyxpearl.com

Comments on this entry are closed.