A number of people testified, including my colleague Doug Brent, but it is the testimony of Steve DelBianco I found particularly intriguing. His testimony revolved around the notion the country-code top-level domains are “controlled by governments”, and future IDN fast track ccTLD allocations will be “reserved only for governments”.

I think many in the ccTLD community will be puzzled by these repeated assertions in his testimony.

Let’s set the stage a little. Country-code top-level domains have existed since the mid-1980s — they are the domains that currently end with two-letter extensions like .FI for Finland, and .DE for Germany. Each country has one available for their use, taken from the ISO 3166-1 standard, but at present they are all written in the letters used for English, known as Latin characters. One of ICANN’s key current initiatives is to work on allowing country-codes to be deployed in different scripts, such as those used for Chinese, Russian and Arabic languages. It is not terribly convenient for those who type in these languages to have to switch their computer to using Latin characters just to put the two-letter endings on their domains, and this will address that.

Recognising that coming up with a complete solution for these internationalised country codes will take some time, the community is working on a “fast track” programme which allows countries that have a demonstrated immediate need to get early access to using these domains. Applications will need to show that the strings they would like to use (like .рф, .日本国 or .ελ) are not contentious, in addition to meeting all the existing eligibility criteria we use for assigning the Latin-based country codes.

So what are the criteria we use today?

The criteria we use in large part revolve around the consensus of “local Internet community” — a sometimes nebulous concept, to be sure, but in essence recognising it is the Internet community as a whole in the country that should decide how their domain is run, not just the Government.

IANA Staff wrote in 1994 that we assign country code top-level domains to trustees that “carry out the necessary responsibilities, and have the ability to do an equitable, just, honest, and competent job”, and have a “duty to serve the community”. “Significantly interested parties in the domain should agree that the designated manager is the appropriate party.”

With respect to national governments, in 1997 we noted that “an additional factor has become very important since [1994]: the desires of the government of the country.  The IANA takes the desires of the government of the country very seriously, and will take them as a major consideration in any transition discussion.” Subsequent to that, the ICANN Governmental Advisory Committee has also made statements regarding this principle.

Clearly national governments have an important role in country-code top-level domains, but that does not translate to controlling them. It is the local Internet community that we look to to provide guidance on how their domains should be run. We expect governments are an important actor in the local Internet community, and that they are involved in the discussion and decision making. But there is a key difference between that, and them exclusively controlling the domain, or having them reserved for the government’s use. If the top-level domain for a particular country is assigned to its government to operate directly, it is because the local Internet community consensus there has decided that is what is appropriate, versus some other alternative.

A basic description of the evaluation criteria we use are provided in the public summary delegation reports we publish on the IANA website (see here for a recent example). ICANN staff have also been working in recent months on improving the public delegation documentation, in anticipation of the launch of the fast track programme. This documentation will better elaborate our existing processes. It is our hope that this will assist prospective applicants for these domains better understand the evaluation criteria when they submit their applications.

We know that Internet communities in a number of countries are already discussing how best to run a potential fast track internationalised domain, so that they can be ready to present their consensus should the programme be launched. Until then, all countries of the world have their two-letter ASCII code and ICANN continues to receive requests to maintain and transfer these domains in accordance with the community’s wishes.

The networks using these officially unallocated addresses are intended to be private, not visible to the global Internet. Nonetheless, their use can be detected when the private parts of networks connect to their public Internet facing connections, such as the connections to their service providers. The addresses leak in e-mail message headers, DNS queries and other random traffic. In some cases, this unofficial use can cause operational problems.

IANA staff has tried to research which /8s are being used in this way. In 2008 we sponsored research by Duane Wessels into which /8s see the most use. He reported on his research at OARC’s DNS Ops meeting and I wrote about it on this blog. Based on this work, we think the /8s with the most unofficial use are:

1, 2, 5, 14, 23, 39, 42, 100, 101, 107, 175 and 176

Duane Wessels’ research was part of a number of presentations that were part of an awareness campaign we worked on. This included talks at network operator groups and articles in industry journals and in some cases discussions with the users of the unallocated space where we could identify them. While we have discussed the issue with some of these network operators, we haven’t been able to speak directly to everyone making unofficial use of this address space because they tend to do so in private networks.

We know that newly allocated IPv4 /8s can be difficult for assigned users to use at first because old filters which block unallocated addresses are slow to be updated with new allocation information. There might be some extra operational difficulties with these particular /8s if unofficial users of the addresses try to communicate with the newly assigned official users of the same addresses.

The longer-term solution to this problem is for network operators to switch to IPv6 and to stop using the unallocated blocks entirely.

Over the next two years we will continue to allocate from these /8s when making allocations to the RIRs. The RIRs use about one /8 per month and so over the next couple of years we know that all of these /8s will be allocated.

One of the more interesting insights involved looking at network diversity. We want top-level domains to keep functioning no matter what is happening — any conceivable disaster shouldn’t knock a top-level domain off-line. One thing we ask is that top-level domains’ name servers be hosted in at least two distinct networks, so it is guarded against a failure (be it a technical failure, or some other business failure event).

Here is a map of all the country-code top-level domains, and one possibly measure of diversity — the number of “autonomous systems” their name servers are hosted in. Countries marked red are reliant on a single network, and if that network failed it could be disastrous for its users without alternatives. Those orange through green have increasing amounts of diversity in the networks that host their name servers:

If we take a minimum of two networks as our baseline requirement, we can look at how TLDs have met this criteria over a period of time — say the last five years:

The blue line shows IPv4 connectivity and it is pretty good, and rather consistent. But if we judge IPv6 connectivity against the same diversity requirement, shown as the green line, not even 50% of ccTLDs have this level of diversity. If we look at TLDs with any IPv6 it is a little better, but there is still about a third of all ccTLDs with no IPv6 connectivity at all!

The good news is the IPv6 trend lines are heading in the right direction, with the growth of IPv6 deployment even accelerating a little recently. Lets hope this continues so that these critical resources are stable not just for existing Internet users, but for future Internet users as well.

How does the DNS work?

The DNS can be considered to be a question-and-answer system. When you type in an address like “icann.org” into a web browser, your computer needs to turn that into a numeric address of the computer hosting that website. To do this, it sends a question over the Internet to a DNS server “Where is icann.org?” The DNS server sends back an answer, “The address is 192.0.2.0”.

How do you attack the DNS?

Let’s say I want to execute an attack on this question and answer exchange, in order to convince the computer to go to the wrong address. When the computer I wish to attack asks a question, my goal is to provide a fake response back to that computer quicker than the real server comes back with the response. By getting my forged answer back faster, the computer will proceed using my answer, rather than the official one.

So, if I send back a fake address to a computer, I can get that computer to go to a different address than the one intended. For example, on that address I might have set up a fake website intended to take someone’s sensitive data, like a replica of a bank’s website.

If I manage to attack one computer why is that a big deal?

A successful attack on one computer in isolation can be problematic for the user of that computer, but it is not that interesting to an attacker to succeed against only one person. Unlucky for us, just one successful attack can very easily have wider consequences. Let me explain.

The DNS is made much more efficient by the use of “caching” name servers. These name servers sit at ISPs, or on corporate networks, and perform DNS lookups on behalf of customers. It then stores the answers it receives in a cache, so for future lookups for the same domain it does not repeat the lookup – it just remembers the previous answer.

This means that if you execute an attack and it gets stored in a cache, it can actually impact many people over and over again, because that answer will be redistributed to everyone that uses that same caching server.

This is why this type of attack is usually called a “cache poisoning” attack, because by poisoning the cache with the wrong data, it creates a much more serious problem.

So I just send back an answer quicker, and that’s it?

It is not quite as simple as just sending a quicker answer back to a computer, you also have to guess certain attributes correctly on the answer that match the question. For example, your answer needs to go back to the same computer the question originated. You also need match the question that was being asked in your answer.

It is simple, however, to guess most of the attributes. As you know which computer you are trying to attack, you don’t need to guess that. As you know which domain you are trying to impersonate, that is also a given. Conventionally, there are only two variables. One variable is you need to guess which server the answer is coming from. The average domain on the Internet has around two or three name servers, only one of which will respond to any given query. Therefore you have about a one in three chance of guessing that correctly. The second variable is a unique reference number (formally, a “transaction ID”), that has about 65000 possibilities. Therefore you have about a 1 in 65000 chance of guessing that correctly.

Earlier this year, security researcher Dan Kaminsky found that it is devastatingly simple to exhaust all those possibilities in a very short amount of time by performing an attack in a certain way. How short? Well, British DNS researcher John Dickinson did some tests and found that on average he could successfully attack a server in just 1.3 seconds.

How do you fix the problem?

The sad news is there is no real solution as far as the regular DNS is concerned. It is not like a security hole in a piece of software that can be repaired with an update. This is an architectural flaw in the DNS protocol itself. There are patches for DNS software, but these only attempt to make executing an attack more difficult, they don’t solve the problem.

Some of the short term approaches to make attacks more difficult are as follows:

1. Randomise the “source port”. One of the attributes in the packet that an attacker needs to guess is called the port number. For architectural reasons, this needs to be port 53 on the way to the server — this is how the server realises it is a DNS query as opposed to a different type of query. However, the port number that a response is sent back to doesn’t need to be port 53. By randomising this, you make it harder for an attacker to guess. Much of the software updates to this problem in mid-2008 related to adding source port randomisation.
2. Block open recursive name service. If you provide access to a caching name server to the whole Internet, then it is very easy for the whole Internet to execute an attack against your server. If you limit access to just those who need it (i.e. your local network), then you reduce that risk.
3. Experimentation with capitalisation of domains. Domains in practice are not case sensitive – if you type ICANN.ORG or icann.org, it means the same thing. However, inside the DNS protocol itself, the encoded transmission between computers actually is case sensitive. This property can be used to add some more randomness to transmissions. If my computer sends off a question asking about “iCaNn.OrG” and gets back an answer for “icann.org”, it can throw it away as untrustworthy. This approach is still experimental and being discussed.

The net effect of these attempts to reduce the risk of attacks primarily involve adding more randomness for the attacker to guess. These approaches approximately double the number of “bits” of randomness. To be clear though, they only make an attack harder, but an attack is still viable. Furthermore, we know that both network speeds and computer speeds get faster and faster each year. These are the two things that slow down an attacker. Therefore, we know that successful attacks will just be easier and easier into the future.

If there is no short term solution, what is the long term solution?

While the DNS itself can’t be properly fixed for the security problem, a new protocol that overlays the DNS called DNSSEC does. DNSSEC uses a system of certification to show that a DNS answer has not been modified. If someone tries to execute an attack, the certificate won’t validate, and the incorrect answer will be thrown away.

DNSSEC is difficult to deploy. It requires upgrades in DNS servers, it changes the way domain name holders manage their name servers, and it adds extra complexity. However, with the knowledge that DNS attacks are so simple to execute without it, there is growing consensus that the pain it will take to deploy is less than the pain of a DNS that you can no longer trust.

More information

You can view the presentation slides used in Cairo.

So where is IPv6 deployment most evident? It?s a very difficult thing to measure. It is difficult to measure the amount of IPv6 traffic as so much of it is tunneled inside of IPv4. And anyway, tunneled traffic is probably from end users rather than ISPs, but we need ISPs to deploy IPv6 to allow the Internet to grow. So how can we see where ISPs are deploying IPv6 in their networks?

One possible measure of IPv6 deployment in ISPs is the number of IPv6 address blocks (prefixes) seen in the routing table in comparison with the the number of autonomous systems (ASs – roughly equivalent to ISPs) in a region. Geoff Huston has a regional breakdown of advertised ASs on his web site and the SixXS project has a regional breakdown of the IPv6 address blocks visible per region on its web site.

AfriNIC, the Regional Internet Registry for Africa and parts of the Indian Ocean, has a higher proportion of networks in its region announcing IPv6 addresses than the others. Africa also has a smaller deployed base but IPv6’s size is designed to support exactly the kind of network growth that highly populated areas, like Africa and Asia will see as their deployed base grows in the next few years.

Proportion of ASs in RIPE NCC service region announcing IPv6 prefixes

Proportion of ASs in APNIC service region announcing IPv6 prefixes

Proportion of ASs in ARIN service region announcing IPv6 prefixes

Proportion of ASs in LACNIC service region announcing IPv6 prefixes

Proportion of ASs in AfriNIC service region announcing IPv6 prefixes

As these events occur, ICANN invariably receives requests to recognise new sovereign entities. In some cases we see very inaccurate press reports by “experts” on how country codes will be assigned. Thankfully, we have a very clear process for this that it is worth repeating.

I said in a blog post a couple of years ago the following:

Another thing ICANN is not involved in is deciding the actual codes, or what constitutes a country eligible for a code. Valid country codes are defined by the ISO 3166-1 standard, which is used internationally not just for domain names — but for physical mail routing, currency codes, and more. The ISO 3166 Maintenance Agency is responsible for keeping the list of codes up to date, taking advice from the United Nations Statistics Office on what constitutes a country eligible for a country code.

By ensuring ICANN is not tasked with deciding what country codes are valid, ICANN can focus its coordination role by ensuring the country-code domains in the DNS root zone match those allowed by the ISO standard. When new countries are formed, new ISO codes are created, and ultimately they can be added as new country code domains. Similarly, countries disappear, their codes are revoked, and they are retired from the DNS root zone.

It is as true today as it was when this policy was introduced in the mid-1980s. We have a more formal description up online, but fundamentally recognition of a new entity that might be granted a country code originates with recognition by the United Nations. Once that occurs, it will kick off a chain of events that will see a new two letter code added to the ISO 3166-1 standard. Once it is in that standard, IANA will accept applications from suitably qualified candidates to operate the country code domain (see our delegation process described here).

This is what happened most recently in the case of Montenegro. In June 2006 it declared independence, was recognised by the United Nations, and added to the ISO 3166-1 standard in September 2006. Some time after this, the Government of Montenegro approached ICANN for delegation, and once the formal process was concluded, .ME was added to the DNS root zone in September 2007.

As at this time, Abkhazia, Kosovo, Transnistria, Somaliland, South Ossetia and others are not in the ISO 3166-1 standard, so ICANN is not in a position to grant any corresponding country-code domain for them. By strictly adhering to the ISO 3166-1 standard, we ensure that ICANN remains neutral by relying upon a widely recognised and impartial international standard.

Duane Wessels of The Measurement Factory analysed DNS queries for evidence of how unallocated IPv4 addresses are being used and presented them at the last OARC meeting, last month. The results cannot give a complete view of what is happening, as it does not see what is happening in private, behind closed firewalls. Nonetheless, these are useful data.

We will be using these data to help plan for the last few IPv4 allocations to the RIRs.

Unfortunately, even when production quality IPv6 transport and network infrastructure are available it is not always possible to deploy a completely IPv6 accessible network. One problem is the difficulties domain name registrants have when they ask their domain name registrar to include their IPv6 glue in the DNS. Not many domain name registrars support glue registration for IPv6 addresses. This limits their ability to provide an IPv6 DNS service.

The problem was discussed during Registries and Registrars’ IPv6 workshop on the last day of the ICANN meeting in Paris. Raúl Echeberría explained the problems that LACNIC has experienced in registering the glue they need for ns.lacnic.net.

Mohsen Souissi of AFNIC then explained that IPv6 support in domain name registries is no longer the hard work it once was. Most of the tools that are needed already support IPv6 very well and have done so for some time. He was followed by Jean-Claude Michot of BookMyName who explained that introducing support for IPv6 glue was not a complicated process and was done very quickly.

It is possible for a registrar to provide support for IPv6 glue registration without running IPv6 on their network at all and deploying an IPv6 network is now far less painful than it once was. Michele Neylon from Blacknight described a generally positive experience.

We hope that more domain registrars will start offering IPv6 glue registration, which will make it easier for domain name registrants to go ahead and deploy their own IPv6 networks.

More information on this event is in the UNDP’s press release of the event, as well as of course the IANA delegation report.

At ICANN and for those who were aware of the situation this raised some eyebrows. Immediately after we turned off the old DNS server and noticed queries for the old address kept getting answers, we began looking into the issue by checking the DNS responses, reviewing routing system logs (including asking folks at Renesys and others for assistance), doing traceroutes from various points in the Internet, etc. Our investigation determined that the answers being provided by the machines now answering for the old L-root address were correct, returning proper referrals and answers including returning the correct address (199.7.83.42) for L-root, and as such, no immediate damage was being done. However, those answers should not have been returned period.

Why did the old L-root address keep answering?

As is well known, there are 13 root server IP addresses (and many DNS servers behind those addresses), referenced by a single letter (A through M) in the domain ROOT-SERVERS.NET. ICANN has operated L-root since 1997 or so. As part of an ongoing effort to improve L-root service, and after an extensive community consultation process, ICANN obtained a “critical infrastructure” block of addresses from ARIN and renumbered L-root out of a block that was originally intended for use at exchange points into the new block. Back in October 2007, ICANN announced this renumbering and continued to provide root name service on the old address for L-root while turning up a much beefier service on the new address. As part of the announced renumbering process, ICANN continued to answer DNS queries on the old address for 6 months, at which time root DNS service on 198.32.64.12 would cease.

On May 2, the 6 month clock expired, and we turned off root DNS service at the old address as planned. To our surprise, however, the registrant for the Exchange Point block, Bill Manning of “EP.NET, LLC.” (who under some relationship with the University of Southern California, Information Sciences Institute helps operate B.ROOT-SERVERS.NET) had entered into an agreement with a DNS hosting provider, CommunityDNS, to continue answering DNS root queries for the old L-root address.

As the folks at Renesys note, Bill Manning, as the current registrant for the superblock out of which for historical reasons the old L-root address was numbered, has the ability to re-provision the 198.32.64.0/24 address space as part of the Exchange Point block of address space. However, the continued availability of DNS root service on a “decommissioned” root server address violates the “Principle of Least Surprise”, at least for those who take a keen interest in the DNS infrastructure such as ICANN. Historically, when a root server has been renumbered, the root server operator maintains a DNS server at the old address for some period of time and then turns off service, with the address not being put back into use. The fact that queries to the old L-root address continued to be answered and ICANN, the operator of the L-root server, was unaware of agreements to continue service by a third party raises some significant areas of concern, not least of which is what should be done with IP addresses formerly used by root servers and who has the authority to make those sorts of decisions.

What did ICANN do?

After some discussion with the relevant parties, the DNS root service being provided at the old L-root address was discontinued. While undoubtedly a number of DNS queries were (and are) still being sent to the old L-root IP address, it is important to remember that the nature of the queries. When a typical DNS resolver first starts up, it uses information called the “root hints” as a list of known IP addresses for root servers. The DNS resolver will then ask one of those IP addresses (typically chosen at random) for an updated list, and then use that updated list from then on. This initial query is known as a “priming query” and only a fraction (1/13th to be precise) of priming queries from resolvers that had not updated their root hints file would have been affected — any queries following that priming query would not be affected since the answer to the priming query returns the correct root server address list.

ICANN has also been monitoring the results returned by these IP addresses through the entire time it was advertised, and believes it was always providing accurate root responses throughout its existence. ICANN continues to work with the root server operator community to improve monitoring and analysis of the root server system, aiming to ensure the continued security and stability of this critical component of the Internet.

So what does this mean for the Internet?

This incident highlights firstly how robust the DNS system is, that it continued to work flawlessly despite this issue. Given the existence of 13 root server IP addresses, only 1/13th of DNS resolvers that have not updated their root hints would have sent a priming query to the IP address formerly associated with the L-root server. However this incident also highlights how important it is for system administrators and DNS software developers to update their root hints as it becomes necessary. Thankfully, this usually only happens once every few years, and IANA sends announcements to the network operator communities when this information is updated.

ICANN, though its Root Server System Advisory Committee (RSSAC), is bringing some of the questions and issues raised by this incident to the root server community, who undoubtedly will want to consider this incident carefully.