The problem is that the unofficial usage makes it more difficult for ISPs to bring these addresses into use when they are officially allocated and so less desirable. But we have to allocate IPv4 addresses to the RIRs as long as we still have them and they still request them. We just need to implement a mechanism to select which /8 is allocated to which RIR.

The mechanism we have implemented reserves two of the /8s showing the least unofficial use for each of the newest RIRs. AfriNIC and LACNIC have fewest IPv4 /8s and service the regions with the most developing economies. We decided that those RIRs should have four of the easiest to use /8s reserved for them.

The other /8s are split between two pools and when APNIC, ARIN or the RIPE NCC qualify for additional IPv4 address space they will be allocated one /8 from each pool, with the /8 being chosen using a verifiable random selection mechanism. The mechanism is based on the “Publicly Verifiable Nomcom Random Selection” mechanism described in RFC 2777.

The sources of randomness used are the prices of the FTSE 100, Dow Jones Industrial Average and Hang Seng Index from midday at the exchange site the day after the request is received, as published on the Yahoo! Finance web site.

The pool of /8s which have been used to number IP networks in an unofficial and improper way is larger than the other pool. So when the smaller pool runs out, all allocations to APNIC, ARIN and the RIPE NCC will come from the larger pool until it too is empty. Then, if any of the /8s reserved for AfriNIC and LACNIC have not been allocated they will become part of a single pool used for all RIRs.

Of course, when all of this is done, there are still five /8s set aside for the implementation of the Global Policy for the Allocation of the Remaining IPv4 Address Space. Very little use of those /8s was detected in our 2008 research.

The networks using these officially unallocated addresses are intended to be private, not visible to the global Internet. Nonetheless, their use can be detected when the private parts of networks connect to their public Internet facing connections, such as the connections to their service providers. The addresses leak in e-mail message headers, DNS queries and other random traffic. In some cases, this unofficial use can cause operational problems.

IANA staff has tried to research which /8s are being used in this way. In 2008 we sponsored research by Duane Wessels into which /8s see the most use. He reported on his research at OARC’s DNS Ops meeting and I wrote about it on this blog. Based on this work, we think the /8s with the most unofficial use are:

1, 2, 5, 14, 23, 39, 42, 100, 101, 107, 175 and 176

Duane Wessels’ research was part of a number of presentations that were part of an awareness campaign we worked on. This included talks at network operator groups and articles in industry journals and in some cases discussions with the users of the unallocated space where we could identify them. While we have discussed the issue with some of these network operators, we haven’t been able to speak directly to everyone making unofficial use of this address space because they tend to do so in private networks.

We know that newly allocated IPv4 /8s can be difficult for assigned users to use at first because old filters which block unallocated addresses are slow to be updated with new allocation information. There might be some extra operational difficulties with these particular /8s if unofficial users of the addresses try to communicate with the newly assigned official users of the same addresses.

The longer-term solution to this problem is for network operators to switch to IPv6 and to stop using the unallocated blocks entirely.

Over the next two years we will continue to allocate from these /8s when making allocations to the RIRs. The RIRs use about one /8 per month and so over the next couple of years we know that all of these /8s will be allocated.

One of the more interesting insights involved looking at network diversity. We want top-level domains to keep functioning no matter what is happening — any conceivable disaster shouldn’t knock a top-level domain off-line. One thing we ask is that top-level domains’ name servers be hosted in at least two distinct networks, so it is guarded against a failure (be it a technical failure, or some other business failure event).

Here is a map of all the country-code top-level domains, and one possibly measure of diversity — the number of “autonomous systems” their name servers are hosted in. Countries marked red are reliant on a single network, and if that network failed it could be disastrous for its users without alternatives. Those orange through green have increasing amounts of diversity in the networks that host their name servers:

If we take a minimum of two networks as our baseline requirement, we can look at how TLDs have met this criteria over a period of time — say the last five years:

The blue line shows IPv4 connectivity and it is pretty good, and rather consistent. But if we judge IPv6 connectivity against the same diversity requirement, shown as the green line, not even 50% of ccTLDs have this level of diversity. If we look at TLDs with any IPv6 it is a little better, but there is still about a third of all ccTLDs with no IPv6 connectivity at all!

The good news is the IPv6 trend lines are heading in the right direction, with the growth of IPv6 deployment even accelerating a little recently. Lets hope this continues so that these critical resources are stable not just for existing Internet users, but for future Internet users as well.

As has been discussed a lot lately, the DNS does not have much in the way in inherent security mechanisms. DNSSEC is a newer technology designed to remedy that by adding a layer of cryptographic verification to the DNS. By using DNSSEC, DNS data can be checked and verified to make sure it has not been tampered with in transit over the unprotected Internet.

Key to deploying DNSSEC is deploying it at the root zone level. The root zone is the upper most level in the DNS hierarchy, and is managed under a complex arrangement involving not only ourselves, but also VeriSign and the US Government. Right now, consultations are being made on how best to secure the root zone using DNSSEC, and that discussion is expected to carry on for some time. It is a somewhat political debate, as well as a technical discussion on how to maintain the robustness of a service that is the cornerstone of Internet stability.

The community has recognised that discussion will undoubtedly carry on for some time, but that there is an immediate need to support nascent DNSSEC deployment efforts. To do this a trust anchor repository was proposed, with ICANN requested to operate the service. A trust anchor repository would be a place to hold the security information that would be in the root zone if it were signed. For example, the Swedish country code top-level domain .SE has already implemented DNSSEC, and their trust anchors can be found in the repository. This allows for early adopters who have suitably configured DNSSEC software to obtain that security information independent of the DNS, without waiting for the root zone to have DNSSEC implemented.

Today we have released the first public version of the trust anchor repository after some initial experimentation with some of the core DNSSEC engineering community. We have prepended the word “interim” to its name, just to emphasise that this isn’t permanent, and is only designed to be a stepping stone to the ultimate goal of a DNSSEC-signed root zone.

We do not recommend it for use other than by expert administrators. It is experimental and requires some understanding of DNSSEC to be helpful. We think it will be useful in giving everyone involved better operational experience with DNSSEC, as well as being a helpful nudge on the way toward more universal DNSSEC deployment on the Internet. As a temporary solution, it has its caveats, and we recommend not treating it as an ultimate solution. But with that in mind, we look forward to those who are feeling adventurous to give it a try and provide us with feedback on how we can improve the service.

L.root-servers.net, operated by ICANN, became the seventh of the root servers to have it’s IPv6 address records (AAAA) added into the DNS root-zone. The addition of IPv6 service is part of ICANN’s ongoing commitment to act as a leader in enabling IPv6 services throughout the DNS.

The new IPv6 address is 2001:500:3::42

To access this service DNS operators should update their hints files. In fact checking that you have the latest hints file on a regular basis is always good operational practice.

The latest hints files are available at the following URLs:

ftp://rs.internic.net/domain/named.root
ftp://ftp.internic.net/domain/named.root

The root servers currently answering on IPv6 are:

A.ROOT-SERVERS.NET 2001:503:ba3e::2:30
F.ROOT-SERVERS.NET 2001:500:2f::f
H.ROOT-SERVERS.NET 2001:500:1::803f:235
J.ROOT-SERVERS.NET 2001:503:c27::2:30
K.ROOT-SERVERS.NET 2001:7fd::1
L.ROOT-SERVERS.NET 2001:500:3::42
M.ROOT-SERVERS.NET 2001:dc3::35

How does the DNS work?

The DNS can be considered to be a question-and-answer system. When you type in an address like “icann.org” into a web browser, your computer needs to turn that into a numeric address of the computer hosting that website. To do this, it sends a question over the Internet to a DNS server “Where is icann.org?” The DNS server sends back an answer, “The address is 192.0.2.0”.

How do you attack the DNS?

Let’s say I want to execute an attack on this question and answer exchange, in order to convince the computer to go to the wrong address. When the computer I wish to attack asks a question, my goal is to provide a fake response back to that computer quicker than the real server comes back with the response. By getting my forged answer back faster, the computer will proceed using my answer, rather than the official one.

So, if I send back a fake address to a computer, I can get that computer to go to a different address than the one intended. For example, on that address I might have set up a fake website intended to take someone’s sensitive data, like a replica of a bank’s website.

If I manage to attack one computer why is that a big deal?

A successful attack on one computer in isolation can be problematic for the user of that computer, but it is not that interesting to an attacker to succeed against only one person. Unlucky for us, just one successful attack can very easily have wider consequences. Let me explain.

The DNS is made much more efficient by the use of “caching” name servers. These name servers sit at ISPs, or on corporate networks, and perform DNS lookups on behalf of customers. It then stores the answers it receives in a cache, so for future lookups for the same domain it does not repeat the lookup – it just remembers the previous answer.

This means that if you execute an attack and it gets stored in a cache, it can actually impact many people over and over again, because that answer will be redistributed to everyone that uses that same caching server.

This is why this type of attack is usually called a “cache poisoning” attack, because by poisoning the cache with the wrong data, it creates a much more serious problem.

So I just send back an answer quicker, and that’s it?

It is not quite as simple as just sending a quicker answer back to a computer, you also have to guess certain attributes correctly on the answer that match the question. For example, your answer needs to go back to the same computer the question originated. You also need match the question that was being asked in your answer.

It is simple, however, to guess most of the attributes. As you know which computer you are trying to attack, you don’t need to guess that. As you know which domain you are trying to impersonate, that is also a given. Conventionally, there are only two variables. One variable is you need to guess which server the answer is coming from. The average domain on the Internet has around two or three name servers, only one of which will respond to any given query. Therefore you have about a one in three chance of guessing that correctly. The second variable is a unique reference number (formally, a “transaction ID”), that has about 65000 possibilities. Therefore you have about a 1 in 65000 chance of guessing that correctly.

Earlier this year, security researcher Dan Kaminsky found that it is devastatingly simple to exhaust all those possibilities in a very short amount of time by performing an attack in a certain way. How short? Well, British DNS researcher John Dickinson did some tests and found that on average he could successfully attack a server in just 1.3 seconds.

How do you fix the problem?

The sad news is there is no real solution as far as the regular DNS is concerned. It is not like a security hole in a piece of software that can be repaired with an update. This is an architectural flaw in the DNS protocol itself. There are patches for DNS software, but these only attempt to make executing an attack more difficult, they don’t solve the problem.

Some of the short term approaches to make attacks more difficult are as follows:

1. Randomise the “source port”. One of the attributes in the packet that an attacker needs to guess is called the port number. For architectural reasons, this needs to be port 53 on the way to the server — this is how the server realises it is a DNS query as opposed to a different type of query. However, the port number that a response is sent back to doesn’t need to be port 53. By randomising this, you make it harder for an attacker to guess. Much of the software updates to this problem in mid-2008 related to adding source port randomisation.
2. Block open recursive name service. If you provide access to a caching name server to the whole Internet, then it is very easy for the whole Internet to execute an attack against your server. If you limit access to just those who need it (i.e. your local network), then you reduce that risk.
3. Experimentation with capitalisation of domains. Domains in practice are not case sensitive – if you type ICANN.ORG or icann.org, it means the same thing. However, inside the DNS protocol itself, the encoded transmission between computers actually is case sensitive. This property can be used to add some more randomness to transmissions. If my computer sends off a question asking about “iCaNn.OrG” and gets back an answer for “icann.org”, it can throw it away as untrustworthy. This approach is still experimental and being discussed.

The net effect of these attempts to reduce the risk of attacks primarily involve adding more randomness for the attacker to guess. These approaches approximately double the number of “bits” of randomness. To be clear though, they only make an attack harder, but an attack is still viable. Furthermore, we know that both network speeds and computer speeds get faster and faster each year. These are the two things that slow down an attacker. Therefore, we know that successful attacks will just be easier and easier into the future.

If there is no short term solution, what is the long term solution?

While the DNS itself can’t be properly fixed for the security problem, a new protocol that overlays the DNS called DNSSEC does. DNSSEC uses a system of certification to show that a DNS answer has not been modified. If someone tries to execute an attack, the certificate won’t validate, and the incorrect answer will be thrown away.

DNSSEC is difficult to deploy. It requires upgrades in DNS servers, it changes the way domain name holders manage their name servers, and it adds extra complexity. However, with the knowledge that DNS attacks are so simple to execute without it, there is growing consensus that the pain it will take to deploy is less than the pain of a DNS that you can no longer trust.

More information

You can view the presentation slides used in Cairo.

So where is IPv6 deployment most evident? It?s a very difficult thing to measure. It is difficult to measure the amount of IPv6 traffic as so much of it is tunneled inside of IPv4. And anyway, tunneled traffic is probably from end users rather than ISPs, but we need ISPs to deploy IPv6 to allow the Internet to grow. So how can we see where ISPs are deploying IPv6 in their networks?

One possible measure of IPv6 deployment in ISPs is the number of IPv6 address blocks (prefixes) seen in the routing table in comparison with the the number of autonomous systems (ASs – roughly equivalent to ISPs) in a region. Geoff Huston has a regional breakdown of advertised ASs on his web site and the SixXS project has a regional breakdown of the IPv6 address blocks visible per region on its web site.

AfriNIC, the Regional Internet Registry for Africa and parts of the Indian Ocean, has a higher proportion of networks in its region announcing IPv6 addresses than the others. Africa also has a smaller deployed base but IPv6’s size is designed to support exactly the kind of network growth that highly populated areas, like Africa and Asia will see as their deployed base grows in the next few years.

Proportion of ASs in RIPE NCC service region announcing IPv6 prefixes

Proportion of ASs in APNIC service region announcing IPv6 prefixes

Proportion of ASs in ARIN service region announcing IPv6 prefixes

Proportion of ASs in LACNIC service region announcing IPv6 prefixes

Proportion of ASs in AfriNIC service region announcing IPv6 prefixes

Duane Wessels of The Measurement Factory analysed DNS queries for evidence of how unallocated IPv4 addresses are being used and presented them at the last OARC meeting, last month. The results cannot give a complete view of what is happening, as it does not see what is happening in private, behind closed firewalls. Nonetheless, these are useful data.

We will be using these data to help plan for the last few IPv4 allocations to the RIRs.

Unfortunately, even when production quality IPv6 transport and network infrastructure are available it is not always possible to deploy a completely IPv6 accessible network. One problem is the difficulties domain name registrants have when they ask their domain name registrar to include their IPv6 glue in the DNS. Not many domain name registrars support glue registration for IPv6 addresses. This limits their ability to provide an IPv6 DNS service.

The problem was discussed during Registries and Registrars’ IPv6 workshop on the last day of the ICANN meeting in Paris. Raúl Echeberría explained the problems that LACNIC has experienced in registering the glue they need for ns.lacnic.net.

Mohsen Souissi of AFNIC then explained that IPv6 support in domain name registries is no longer the hard work it once was. Most of the tools that are needed already support IPv6 very well and have done so for some time. He was followed by Jean-Claude Michot of BookMyName who explained that introducing support for IPv6 glue was not a complicated process and was done very quickly.

It is possible for a registrar to provide support for IPv6 glue registration without running IPv6 on their network at all and deploying an IPv6 network is now far less painful than it once was. Michele Neylon from Blacknight described a generally positive experience.

We hope that more domain registrars will start offering IPv6 glue registration, which will make it easier for domain name registrants to go ahead and deploy their own IPv6 networks.

At ICANN and for those who were aware of the situation this raised some eyebrows. Immediately after we turned off the old DNS server and noticed queries for the old address kept getting answers, we began looking into the issue by checking the DNS responses, reviewing routing system logs (including asking folks at Renesys and others for assistance), doing traceroutes from various points in the Internet, etc. Our investigation determined that the answers being provided by the machines now answering for the old L-root address were correct, returning proper referrals and answers including returning the correct address (199.7.83.42) for L-root, and as such, no immediate damage was being done. However, those answers should not have been returned period.

Why did the old L-root address keep answering?

As is well known, there are 13 root server IP addresses (and many DNS servers behind those addresses), referenced by a single letter (A through M) in the domain ROOT-SERVERS.NET. ICANN has operated L-root since 1997 or so. As part of an ongoing effort to improve L-root service, and after an extensive community consultation process, ICANN obtained a “critical infrastructure” block of addresses from ARIN and renumbered L-root out of a block that was originally intended for use at exchange points into the new block. Back in October 2007, ICANN announced this renumbering and continued to provide root name service on the old address for L-root while turning up a much beefier service on the new address. As part of the announced renumbering process, ICANN continued to answer DNS queries on the old address for 6 months, at which time root DNS service on 198.32.64.12 would cease.

On May 2, the 6 month clock expired, and we turned off root DNS service at the old address as planned. To our surprise, however, the registrant for the Exchange Point block, Bill Manning of “EP.NET, LLC.” (who under some relationship with the University of Southern California, Information Sciences Institute helps operate B.ROOT-SERVERS.NET) had entered into an agreement with a DNS hosting provider, CommunityDNS, to continue answering DNS root queries for the old L-root address.

As the folks at Renesys note, Bill Manning, as the current registrant for the superblock out of which for historical reasons the old L-root address was numbered, has the ability to re-provision the 198.32.64.0/24 address space as part of the Exchange Point block of address space. However, the continued availability of DNS root service on a “decommissioned” root server address violates the “Principle of Least Surprise”, at least for those who take a keen interest in the DNS infrastructure such as ICANN. Historically, when a root server has been renumbered, the root server operator maintains a DNS server at the old address for some period of time and then turns off service, with the address not being put back into use. The fact that queries to the old L-root address continued to be answered and ICANN, the operator of the L-root server, was unaware of agreements to continue service by a third party raises some significant areas of concern, not least of which is what should be done with IP addresses formerly used by root servers and who has the authority to make those sorts of decisions.

What did ICANN do?

After some discussion with the relevant parties, the DNS root service being provided at the old L-root address was discontinued. While undoubtedly a number of DNS queries were (and are) still being sent to the old L-root IP address, it is important to remember that the nature of the queries. When a typical DNS resolver first starts up, it uses information called the “root hints” as a list of known IP addresses for root servers. The DNS resolver will then ask one of those IP addresses (typically chosen at random) for an updated list, and then use that updated list from then on. This initial query is known as a “priming query” and only a fraction (1/13th to be precise) of priming queries from resolvers that had not updated their root hints file would have been affected — any queries following that priming query would not be affected since the answer to the priming query returns the correct root server address list.

ICANN has also been monitoring the results returned by these IP addresses through the entire time it was advertised, and believes it was always providing accurate root responses throughout its existence. ICANN continues to work with the root server operator community to improve monitoring and analysis of the root server system, aiming to ensure the continued security and stability of this critical component of the Internet.

So what does this mean for the Internet?

This incident highlights firstly how robust the DNS system is, that it continued to work flawlessly despite this issue. Given the existence of 13 root server IP addresses, only 1/13th of DNS resolvers that have not updated their root hints would have sent a priming query to the IP address formerly associated with the L-root server. However this incident also highlights how important it is for system administrators and DNS software developers to update their root hints as it becomes necessary. Thankfully, this usually only happens once every few years, and IANA sends announcements to the network operator communities when this information is updated.

ICANN, though its Root Server System Advisory Committee (RSSAC), is bringing some of the questions and issues raised by this incident to the root server community, who undoubtedly will want to consider this incident carefully.