Renesys Blog: Pakistan hijacks YouTube

Pakistan hijacks YouTube

Late in the (UTC) day on 24 February 2008, Pakistan Telecom (AS 17557) began advertising a small part of YouTube's (AS 36561) assigned network. This story is almost as old as BGP. Old hands will recognize this as, fundamentally, the same problem as the infamous AS 7007 from 1997, a more recent ConEd mistake of early 2006 and even TTNet's Christmas Eve gift 2004.

Just before 18:48 UTC, Pakistan Telecom, in response to government order to block access to YouTube (see news item) started advertising a route for 208.65.153.0/24 to its provider, PCCW (AS 3491). For those unfamiliar with BGP, this is a more specific route than the ones used by YouTube (208.65.152.0/22), and therefore most routers would choose to send traffic to Pakistan Telecom for this slice of YouTube's network.

I became interested in this immediately as I was concerned that I wouldn't be able to spend my evening watching imbecilic videos of cats doing foolish things (even for a cat). Then, I started to examine our mountains of BGP data and quickly noticed that the correct AS path ("Will the real YouTube please stand up?") was getting restored to most of our peers.

The data points identified below are culled from over 250 peering sessions with 170 unique ASNs. While it is hard to describe exactly how widely this hijacked prefix was seen, we estimate that it was seen by a bit more than two-thirds of the Internet.

This table shows the timing of the event and how quickly the route propagated (this is actually a fairly normal propagation pattern). The ASNs seeing the prefix were mostly transit ASNs below, so this means that these routes were distributed broadly across the Internet. Almost all of the default free zone (DFZ) carried the hijacked route at least briefly.

18:47:00uninterrupted videos of exploding jello
18:47:45first evidence of hijacked route propagating in Asia, AS path 3491 17557
18:48:00several big trans-Pacific providers carrying hijacked route (9 ASNs)
18:48:30several DFZ providers now carrying the bad route (and 47 ASNs)
18:49:00most of the DFZ now carrying the bad route (and 93 ASNs)
18:49:30all providers who will carry the hijacked route have it (total 97 ASNs)
20:07:25YouTube, AS 36561 advertises the /24 that has been hijacked to its providers
20:07:30several DFZ providers stop carrying the erroneous route
20:08:00many downstream providers also drop the bad route
20:08:30and a total of 40 some-odd providers have stopped using the hijacked route
20:18:43and now, two more specific /25 routes are first seen from 36561
20:19:3725 more providers prefer the /25 routes from 36561
20:28:12peers of 36561 start seeing the routes that were advertised to transit at 20:07
20:50:59evidence of attempted prepending, AS path was 3491 17557 17557
20:59:39hijacked prefix is withdrawn by 3491, who disconnect 17557
21:00:00the world rejoices; Leeroy Jenkins online again.

Since BGP relies on a transitive trust model, validation between customer and provider is important. In this case, PCCW (3491) did not validate Pakistan Telecom's (17557) advertisement for 208.65.153.0/24. By accepting this advertisement and readvertising to its peers and providers PCCW was propagating the wrong route. Those who saw this route from PCCW selected it since it was a more specific route. YouTube was advertising 208.65.152.0/22 before the event started and the /24 was a smaller (and more specific) advertisement. According to usual BGP route selection process, the /24 was then chosen, effectively completing the hijack.

Because of the fast detection and reaction of the YouTube staff and cooperation with other providers, service for their (sub-) prefix was interrupted for about an hour and forty minutes for some lucky customers and, at most, a bit more than two hours. The exact duration of the outage depends on your vantage point on the Internet.

When these sorts of events occur, there is renewed interest in a variety of solutions to this problem. BGP is fundamental to provider relationships and will not be going away anytime soon. Cryptographic extensions to BGP have been suggested (Pretty Good BGP, Secure Origin BGP and SBGP). These may be too taxing for router CPUs. Of course, after any sort of hijacking event (whether inadvertent or malicious) prefix and AS monitoring is suggested (e.g., the Internet Alert Registry, the Prefix Hijack Alert System, RIPE's MyASN and Renesys' Routing Intelligence).

Ultimately, though, the problem remains one of transitive trust. A provider can and should limit the advertisements it will accept from a customer. The mechanics can be arranged manually or can be configured using Routing Policy Specification Language (RPSL) to communicate the policy and drive configuration. In the case of Pakistan Telecom, they originate or transit fewer than 1000 prefixes.

So, it's heartwarming to know that two things are still true. It is still trivially possible to hijack prefixes (whether maliciously or inadvertently). I can go to sleep knowing that my neighbors are happily watching their LOLCATS.

TrackBack

Listed below are links to weblogs that reference Pakistan hijacks YouTube:

» Pakistan hijacks YouTube from exact
Bookmarked your post over at Blog Bookmarker.com! [Read More]

» Pakistan blocks YouTube, breaks trust from billso.com
Earlier today, we noticed that YouTube was not available. An ISP in Pakistan, PieNet, single-handedly blocked global access to the popular video site for two hours, according to multiple reports on the Times of London, ZDnet, ReneSys, OpenDNS and Data ... [Read More]

» Ring! Ring! Hot News, 25th February 2008 from Telco 2.0
Concentrated links, every Monday. [Read More]

» YouTube Offline, Pakistan Telecom Blamed from Data Center Knowledge
YouTube was offline for about two hours Sunday, sparking a debate about whether the outage was caused by an effort by Pakistan to block the site. [Read More]

» How to Avoid Another Major IP Hijacking from Data Center Knowledge
YouTube isn't the first site to have its IP space hijacked. Some history, and a look at existing preventive measures. [Read More]

» The fragility of the Internets - as demonstrated by Pakistan / Youtube from Robert Hensing's Blog
I love how fragile the Internet really is. This is demonstrated from time to time and when it is - I'm [Read More]

» Pakistan Telecom Hijacked Youtube from delusionofgrandeur
It could have been completely accidental but Pakistan Telecom, in trying to comply with a Pakistan government censorship order, hijacked part of Youtube’s internet routing last night. Renesys blog tells us: Just before 18:48 UTC, Pakistan Teleco... [Read More]

» Pakistan hijacks YouTube... [Spyware Sucks] from Australian & New Zealand MVPs
Those of you with a technical mindset may find this explanation about what happened, and the timeline [Read More]

» Pakistani Hijack Of Youtube: The HK Connection from Daai Tou Laam Diary
Going through the RSS feed for the day, it seems that a Pakistani government order to ban Youtube resulted in a temporary hijack of Youtube's internet routing information. (via Wampum) But it seems there is a Hong Kong connection to the hijacking that pro [Read More]

» La ragnatela from gianlucalini.it
ha dei buchi! Scherzi a parte, non so quanti di voi abbiano approfondito l'hijacking dell'address space di YouTube occorso Domenica e tutte le discussioni nate dopo, a questo proposito una buona fonte può essere la mailing list nanog. Ho creato un elenc [Read More]

» YouTube outage *updated* (caused by routing filter mistake in Pakistan) from HCS's and Gen's Place
youtube had their ip’s hijacked. Pakistan was advertising an invalid route announcement which not only blocked youtube for Pakistan but other networks for some reason accepted this as a valid route and blocked youtube for other networks as well.... [Read More]

» How Pakistan Hijacked YouTube from Glen Bowes
On February 24, 2008 in response to a government order, a Pakistani ISP (Internet Service Provider, a business that provides access to the Internet such as Bell, Cogeco, and IAW) PieNet, began blocking access to a YouTube video that apparently containe... [Read More]

» Der gekidnappte Youtube IP Prefix - das Protokoll from CH Internet Szene
von Fredy K�nzler Die Netzwerk-Analysten von Renesys haben ihren Datenhaufen durchw�hlt und auf ihrem Blog eine genaue Analyse ver�ffentlicht, wie der Youtube-IP-Prefix von Pakistan Telecom gekidnappt worden ist. Oder, salopper ausgedr�ckt: so machen d [Read More]

» Pakistan Blocks YouTube Access from World Views
Last week, the Pakistani ordered all Internet service providers to block the YouTube website for containing [Read More]

» Pakistan Blocks YouTube Access from World Views
Last week, the Pakistani Government ordered all Internet service providers to block the YouTube website [Read More]

Comments

Wait: the order PDF mentions a specific video. It has been deleted due to a "terms of use violation": what was it?

You said that lucky folks only noticed a 30 minute outage. However, in the timeline you posted there is a 1hr20min gap between action and the first reaction. (18.49 to 20.07)

Could you clarify what fixed the bad route problem for any affected parties after about 30 minutes, and when was that countermeasure taken?

I'm hearing rumblings that the block of YouTube had more to do with videos showing how the elections were rigged, and less to do with the "blasphemous" videos. The latter was simply an easy excuse to block the former.

Really nice post, and better than slogging through the NANOG hijack thread.

Thanks, xan!

My explanation did indeed say thirty minutes, though it should have said "about an hour and thirty minutes". (It was actually about 1h42m, but I say an hour and forty minutes in the corrected text above.)

I appreciate the correction!


-Martin

Cryptographic BGP extension can't help in this case. It only tell who announce this, not who can announce this.

The upstream provider (AS 3491) don't filter any route. Just knowing who help nothing

So, in fact it was NOT Pakistan or the Pakistan Telecom Authority that blocked YouTube, but a technician at PCCW who did not verify the PTA's routing advertisement.

I'm no fan of political censorship, and think that trying to prevent people from seeing a cartoon is self-defeating and wrong.

But as a journalist, I am a fan of the truth, which is that PCCW caused the problem, which it can correct by implementing a manual verification procedure before complying with customer requests.

Is that right, Earl?

----------

Earl: If I light my neighbor's house on fire and burn it to the ground, do you place blame solely on the fire department for not seeing the smoke and putting out the fire in time? All providers need to be good net citizens, which including not injecting garbage into the routing tables and also guarding against it from others - when possible. Both parties are responsible, but the source of "the fire" bears the greater responsibility.

And here is an lolcat just for for this occasion...

http://nicklevay.net/misc/bgpcat.jpg

Here's a BGPlay link (using RIPE RIS data) that nicely shows the propagation dynamics for the /24.

http://www.ris.ripe.net/cgi-bin/bgplay.cgi?prefix=208.65.153.0/24&start=2008-02-24+18:46&end=2008-02-24+21:05
--
Simon.

Thank you for this detailed technical account. The mass media accounts have, as usual, been an unintelligible mishmash.

Nathaniel,

The point is that there were two technical errors. First, Pakistan Telecom was advertising a route they had only intended to blackhole. Second, PCCW didn't have prefix filters installed to limit the reach of this advertisement.

Also routing advertisements are usually subject to a series of checks--it's unfortunate that PCCW did not have prefix checks to prevent this entire situation.

-Martin

Just a note: for those who want a lower-latency way to discusss this event, we started a few discussions over at Babbledog, Renesys's personalized social news project.

Babbledog supports live discussion without moderation or waiting for your posts to show up.

Take a look at this this discussion or search for related related stories

Its open again, i can view youtube here in Islamabad

Hmmm... I'm living in China now and here, in Beijing, I often collide with site blocking. To prevent that, I'm using http://strongvpn.com. It’s a VPN account with strong and reliable service. I haven't use a new proxy every day after its blocking.