Policy, Technology

Why Physical Cyber Security is Broken

Leave a comment

Recent FBI reports of a dozen physical attacks on the Internet backbone near Sacramento, CA highlight the sad state of physical cyber security and policies to cope with threats to the Internet. Sacramento is an intermediary in the Internet backbone carrying east-west traffic; it connects Chicago and San Francisco hubs, and ultimately links the West Coast with the remainder of the U.S. and Europe. No wonder the bad guys physically cut the fiber optic cable near Sacramento—it is a critical link in the most highly traveled cables in the U.S.

A popular misconception is the physical Internet is among the most resilient infrastructures in modern society. After all, it was designed in the 1960s to withstand a nuclear attack. This claim is based on the mistaken belief that the physical Internet contains many alternate routes. When a break on one route is sensed, information is simply re-routed through a redundant alternate path. Unfortunately, this is no longer true in general.

What Happened?

Over the past 20 years—since the Telecommunications Act of 1996—the physical resiliency of the Internet has slowly eroded because of bad policy and cost‐benefit economics. The 1996 Act requires Internet service providers (ISPs) to peer-share each other’s networks. While intended to encourage competition, the unintended consequence is a dramatic increase in vulnerability. In fact, more than 10 years ago NSTAC (National Security Telecommunications Advisory Committee) declared the peering facilities—called “carrier hotels”—the number one vulnerability of the communications infrastructure: “The current environment, characterized by the consolidation, concentration, and collocation of telecommunications assets, is the result of regulatory obligations, business imperatives, and technology changes” [1].

Carrier hotels are a dangerous consequence of regulation, with potentially national impact if one or more should fail: “…Loss of specific telecommunications nodes can cause disruption to national missions under certain circumstances” [1].

Second, economics plays a major role in weakening the Internet. As the Internet becomes increasingly commercialized and regulated, it also becomes increasingly efficient and cost-effective. ISPs like AT&T, Level 3, Verizon, Hurricane Electric, Comcast, and Time Warner eliminate redundancy in the name of efficiency and profitability. What is wrong with efficiency? Efficiency translates into lower redundancy. Lower redundancy means more risk. And more risk means more vulnerability to hacks, outages, and exploits.

For the past decade, ISPs have been putting more eggs into fewer baskets—fewer ISPs and fewer cables carry the global load. Even if ISPs don’t reduce the number of paths available in the physical cabling of the Internet, hidden blocking nodes are very difficult to discover without massive computational resources. Blocking nodes are nodes that segment the Internet into disjoint islands, if removed. They are the most critical of all autonomous systems [2].

Figure 1a. The 15 blocking nodes (red and circled) in the 100 most-connected autonomous systems of the 2004 Internet illustrate Internet fragility. http://www.caida.org. Blocking nodes are nodes that segment the Internet into disjoint islands, if removed. The average length of shortest path is 2.4 hops and the maximum length is 5 hops, and there are 134 links. The most-used links are colored magenta and red; the least‐used are gray, blue, and green. Yellow and orange links fall in between.
Figure 1a. The 15 blocking nodes (red and circled) in the 100 most-connected autonomous systems of the 2004 Internet illustrate Internet fragility (http://www.caida.org). Blocking nodes are nodes that segment the Internet into disjoint islands, if removed. The average length of shortest path is 2.4 hops and the maximum length is 5 hops, and there are 134 links. The most-used links are colored magenta and red; the least‐used are gray, blue, and green. Yellow and orange links fall in between.

Figure 1 illustrates the presence of blocking nodes in the 100 most-connected AS-level ISPs of the 2004 Internet [3]. It is impossible to determine if a node is a blocking node simply by looking at Figure 1. Instead, a very time-consuming algorithm is required to determine if a node is a blocking node. The full AS13579, 37448 Internet containing 13,579 autonomous systems (nodes) and 37,448 peering relations (links) contain 2,006 blocking nodes—a result that takes hours of computation to determine. The presence of blocking nodes means the Internet can be severed into disconnected pieces by removing a single node. For the AS13579, 37448 network, this means removal of any one of 2,006 AS nodes can segment the Internet into disjointed and non-communicating sub-networks.

Figure 1b. The de-percolated network of Figure 1a contains the same 100 nodes, of which 20 are blocking nodes, but only 95 links. De‐percolation means removing links that do not separate the network into disjoint islands, if removed. This de‐ percolated network contains the minimum number of links necessary to remain connected. The average length of shortest path is 3.4 hops and the maximum length is 8 hops. Compare these with average of 2.4 hops and maximum of 5 hops in Figure 1a. The most-used OSPF links are colored magenta and red.
Figure 1b. The de-percolated network of Figure 1a contains the same 100 nodes, of which 20 are blocking nodes, but only 95 links. De‐percolation means removing links that do not separate the network into disjoint islands, if removed. This de‐ percolated network contains the minimum number of links necessary to remain connected. The average length of shortest path is 3.4 hops and the maximum length is 8 hops. Compare these with average of 2.4 hops and maximum of 5 hops in Figure 1a. The most-used OSPF links are colored magenta and red.

Figure 1 also illustrates the effect of the OSPF (Open Shortest Path First) routing algorithm on the topology and structure of the physical Internet. Randomly selecting source and destination nodes in Figure 1 and counting the number of packets traveling along the shortest path between source and destination produces a distribution as shown in Figure 1. Only a very small subset of routes is selected by OSPF, because they are the most efficient paths between randomly selected nodes. Figure 1b shows the reduction of links increases path length, making the OSPF policy behave even worse. Conversely, increasing the number of links reduces the number of hops along the shortest paths, but still overloads a few links. Topology is to blame.

Figure 2. The change in IPV6 autonomous systems (AS) degree from 2013 to 2014 shows that higher degree systems increase their degree, while lower-degree systems generally stay the same or decrease degree. This illustrates preferential attachment: Higher degreed systems attract higher connectivity, e.g. higher degree.
Figure 2. The change in IPV6 autonomous systems (AS) degree from 2013 to 2014 shows that higher degree systems increase their degree, while lower-degree systems generally stay the same or decrease degree. This illustrates preferential attachment: Higher degreed systems attract higher connectivity, e.g. higher degree. (Source: http://www.caida.org/research/topology/as_core_network/2014/)

Now, add the efficiency required of expensive backbone networks to the model. Rather than increase number of links, Tier-1 ISPs increase the bandwidth (or number of cables) of fewer links. This creates a self-organizing feedback loop: Fewer links are used by more ISPs, increasing the hub size (degree) of autonomous systems, and creating more demand for bandwidth because of preferential attachment. Figure 2 shows evidence of this feedback loop in action between 2013 and 2014, when CAIDA measured and reported the changing degrees (number of links) of the top 12 backbone providers. The degree of highly connected systems like Hurricane Electric generally increased, while the degree of less‐connected systems stayed the same or decreased. In other words, high bandwidth and high connectivity attracts more use, which attracts more bandwidth and connectivity, which attracts more use. This self-organizing feedback loop leads to fewer, but larger capacity autonomous systems and physical connections (see Figure 3). Fewer links means less resilience.

Figure 3. The gigabit backbone network of Hurricane Electric, one of the largest Tier-1 internet service providers in the United States, is relatively sparse: 22 nodes are connected by 39 links, for an average degree (connectivity) of 3.54 links. The mean degree (average connectivity) of the AS-level Internet is 5.5.
Figure 3. The gigabit backbone network of Hurricane Electric, one of the largest Tier-1 internet service providers in the United States, is relatively sparse: 22 nodes are connected by 39 links, for an average degree (connectivity) of 3.54 links. The mean degree (average connectivity) of the AS-level Internet is 5.5. (Source: http://he.net/HurricaneElectricNetworkMap.pdf)

 

Figure 4. Analysis of the Hurricane Electric backbone with 22 nodes, 39 links, and 3 blocking nodes yields an average path length of 2.37 hops, and maximum length of 4 hops. The most-used links are between San Francisco and Dallas, and San Francisco Los Angeles. This analysis suggests a handful of nodes and links are critically important to holding the Hurricane Electric backbone together.
Figure 4. Analysis of the Hurricane Electric backbone with 22 nodes, 39 links, and 3 blocking nodes yields an average path length of 2.37 hops, and maximum length of 4 hops. The most-used links are between San Francisco and Dallas, and San Francisco Los Angeles. This analysis suggests a handful of nodes and links are critically important to holding the Hurricane Electric backbone together.

The Hurricane Electric physical backbone of Figure 3 was analyzed for critical nodes and links (see Figure 4). As expected, the Hurricane Electric network suffers from the same self‐organizing feedback loop—most OSPF traffic depends on only a few links, and there are three blocking nodes. These are the critical links and nodes of the physical infrastructure. As this network is optimized to carry more traffic on the most‐used OSPF links, vulnerability of the entire network to targeted attacks only becomes worse.

The bottom line: The physical Internet is evolving away from resiliency toward fragility. Think of it as the interstate highway system where a major freeway is removed every few months to cut costs and optimize traffic by routing more and more traffic through the most‐traveled cities and freeway links. Even if more lanes are added to the overloaded roads, the fact that there are fewer roads leads to bigger catastrophic failures when one is blocked.

What to Do?

What should we do? The first step in securing the Internet is to reverse regulatory policy so that it rewards and encourages physical redundancy. The private sector must be motivated to increase redundancy and resiliency even when it costs more. Improvement in cyber security is good for business. Who is not in favor of better security? But, current regulatory policy forces the private sector to do the wrong thing.

The public sector (government) must enact smarter regulation, but also, come up to speed on how the Internet actually works. The idea that the Internet is an open unfettered free market is false. The net neutrality decision recently made by the FCC doesn’t open and free the Internet—rather, it captures and regulates it. The current policy of an FCC-regulated Internet will have the same effect on the Internet as the 1996 Telecommunications Act had on the communications industry. It will increase vulnerability through a number of unintended consequences.

The second remedy is technical. The OSPF routing policy should be replaced by a random routing policy, and network providers should concentrate on the topology of their networks as much as bandwidth. No one knows what the tradeoff is between congestion and randomization. But we do know that resilience is more expensive than living on the edge.

 

References

[1] NSTAC Task Force on Concentration of Assets: Telecom Hotels. National Security Telecommunications Advisory Committee. U.S. Department of Homeland Security. February 12, 2003.

[2] Lewis, T. G. Critical Infrastructure Protection: Defending a Networked Nation 2nd ed. John Wiley & Sons, 2014.

[3] Huffaker, B., Claffy, K.C., Hyun, Y., and Luckie, M. Pv4 and IPv6 AS Core: Visualizing IPv4 and IPv6 Internet Topology at a Macroscopic Scale in 2014. Center for Applied Internet Data Analysis. 2014.

-41