Google local peering starting (slowly)

Google now serves [almost] all traffic to TENET sites directly from either:

• peering connection, or
• some sort of GGC-like nodes [Google Maps, YouTube videos]
  

Name resolution and tracerouting from UCT hosts confirms that all Google services are being served from local [South African] servers. This has huge implications in bandwidth savings, and quality of user interaction because of lower latency for content that is already available on the local datacenter and caches.

Issues
Strangely, while www.youtube.com resolves to a locally-hosted server (64.233.179.100 currently), youtube.com (with no leading www.) resolves to a 208.65.153.xxx address, seemingly in Richmond, VA.

Clarification
The caching nodes [presumably GGC nodes] currently serve only certain services (map tiles, YouTube videos) and are generally co-located within a client network. The other Google services are served from machines located in Cape Town (based on ping times) and that are administratively within a Google-owned IP block.

Further clarification
Assume this is a "small" trial run (limited number of users on TENET, maximum goodwill from helping the poor academics) before a full roll-out. The caching nodes already save TENET up to 45 Mbps international transport! There is no local visibility to Google's ZA IP block from IS's route server.

TENET peering at LINX

More interesting traceroute data [target www.yahoo.com]:

8 v2750-tiger-brie.uni.net.za (155.232.145.226) 2.732 ms
9 unknown.uni.net.za (196.32.209.25) 152.780 ms
10 ge-1-1-0.pat1.the.yahoo.com (195.66.224.129) 154.808 ms
11 so-0-1-0.msr1.ird.yahoo.com (66.196.65.33) 163.932 ms
12 gi-1-1.bas-b1.ird.yahoo.com (87.248.101.1) 164.020 ms
13 f1.us.www.vip.ird.yahoo.com (87.248.113.14) [open] 165.543 ms


The TENET router in London (196.32.209.25) connects to the Yahoo! AS, and the 195.66.224.0/19 router address is a giveaway that it's through LINX.

Testing other networks [target www.facebook.com] yields more data:

9 unknown.uni.net.za (196.32.209.25) 153.603 ms
10 linx.br01.lon1.tfbnw.net (195.66.225.69) 153.336 ms
11 xe-x-x-x.br01.ash1.tfbnw.net (204.15.22.245) 229.906 ms
12 te-13-0.csw06a.ash1.tfbnw.net (204.15.23.55) 233.942 ms
13 www.11.06.ash1.facebook.com (69.63.186.12) [open] 262.342 ms

where hop 10's hostname confirms the LINX connection, and also shows we're still connecting to the Virginia servers.

The LINX looking glass also shows the routing to TENET through AS1299 (Telia) and AS2914 (NTT) and directly.

Traffic graphs confirm significant traffic through LINX


Several other autonomous systems are reachable via LINX peering sessions, including hosting providers Hurricane Electric and LeaseWeb and CDN Limelight Networks [ex YouTube providers, and 2009 Presidential inauguration streamers].

Clearly this is in preparation for the new SEACOM pipe, although TENET is saving on transit requirements.

Google 'local peering' live?

From some traceroute data [target: www.blogger.com]:

9 unknown.uni.net.za (155.232.253.254) 2.341 ms 2.055 ms 2.237 ms
10 64.233.174.40 151.205 ms 151.276 ms 151.457 ms

Note the jump in RTT as we go from the TENET router to the Google router (64.233.160.0/19 belongs to Google).

Compare to traceroute data for a less aggressively-peering company [target: www.microsoft.com]:

8 v2750-tiger-brie.uni.net.za (155.232.145.226) 2.341 ms 2.254 ms 2.337 ms

9 unknown.uni.net.za (196.32.209.25) 153.050 ms 152.938 ms 152.223 ms

Where the next hop is a TENET-controlled router overseas.

So perhaps Google has started 'peering locally', and are transparently tunneling the data overseas to allow IS [and/or other ISPs] to avoid paying for each user's international bandwidth. Of course Google are therefore paying for it, but with the standard international bandwidth pricing feedback damping the usage, they probably aren't seeing a huge explosion in bandwidth demand.

Contrast this to the local YouTube traffic caching, where having locally-available video means that videos load faster: better user experience translates to higher usage, and the demand increases. For the user this is a virtuous circle, but to the various I[C]T[S] departments it must seem more like a vicious circle.

However, for organisations like TENET where the pipe is purchased old-school (i.e. 'per Mbps' rather than 'per GB'), this will be a boon, because up to 40 Mbps [peak] is now being handed-off locally. Interestingly the upstream/outbound traffic, presumably web crawling, is still going through the Ubuntunet point in London.

Some graphs:

The London peering session going [relatively] quiet late Tuesday.

A 'local peering' session [presumably with Google] in Cape Town going live late Tuesday:

And some of the local caching traffic seems strongly correlated (via Mk. 1 eyeball-based peak matching on other graphs) to the local peering session.

The takeaway: Google is probably peering locally, but is trying to avoid flooding ISPs with bandwidth-conversion economic problems.

Google to merge YouTube AS

Peeringdb.com always yields interesting nuggets [login:guest, password:guest] if you watch which companies are performing updates.

YouTube, although a Google company, still maintain a separate AS, but it looks like that is going to change [sourced from the above-linked peering record]:

Notes: Peering will be migrating to Google (AS15169) by EOY 2009.

Many have wondered how long YouTube would peer independently of Google, and many have also speculated on the technical complexity of merging their operations to allow peering via both AS. I assume Google will start announcing routes to AS 36561 (if they haven't already), and YouTube will start pre-pending [padding] their routes to make the Google routes preferable.

YouTube is peering in two exchanges that Google seems to be missing from:

• Equinix Newark
• PAIX Dallas

One could speculate on whether Google will keep these connections active for AS 15169, as they already privately peer at Equinix Dallas and PAIX New York (amusingly this is almost the opposite Company/Location matching).

Speculations on Google's peering timing

Take the following two events:

• Google should start peering locally "soon". [Events seem to have overtaken the original plans for implementation by the end of January, so perhaps mid-February?]
• SEACOM is coming online "in June".

These might seem unremarkable events individually but when viewed together they provoke interesting questions. At the root of all of these questions has to be some speculation regarding the timing of Google's technical entry into this market (ignoring their sales operation) prompted by this observation:
Local Google peering is very close to, but before, the SEACOM launch.

Some obvious simple questions, with possible answers:

• Q: Are Google using SAT-3 or SEACOM?
  
• A: Google is likely to be using SAT-3.
• It's unlikely that SEACOM have any segments finished and ready for operations yet.
• However Google probably don't need a full path from South Africa to London; the 'express path' to Kenya would help operations both in Kenya and in South Africa, so any costs involved in bringing content to Kenya or South Africa could be amortized over both operations.
• Google would also like some redundancy in their paths, especially considering the Egypt-to-France portion will be the responsibility of Egypt Telecom.
  
  
• Q: Does Google have a big pipe?
  
• A: Probably not.
• 10 Gbps IRUs are available on SEACOM, but SAT-3's total capacity is 120 Gbps. Even 1 Gbps is a large chunk of the available capacity.
• Pricing on SAT-3 circuits is also very high, although it'll be under pressure because of the imminent SEACOM launch.
• However, if Google have negotiated a long-term deal with Telkom, they may have secured better pricing.
• Presuming they buy a circuit on SEACOM, one assumes it will be a 10 Gbps IRU.
  
  
• Q: Why hasn't Google peered in South Africa before this?
• A: It was too expensive for the possible benefit to Google.
• ISPs must secure bandwidth to provide service to their customers. Those customers want connection to Google, so ISPs wanting to keep customers happy bought the bandwidth.
• Google started local [service / sales] operations a while back (witness the recent Yellow Pages / Entelligence / Google spat) and are ramping up their presence in the country. Further usage of YouTube, Google Earth & Google Maps, GMail and other bandwidth-intensive services would help drive traffic (and hence advertising earnings), and usage of Google Docs would help steal marketshare from Microsoft in a developing market that Microsoft itself recognises is vulnerable (hence Microsoft's pre-paid Office offering).
• Google can afford to run the service at a high marginal cost for a while, until SEACOM comes online.
• Also, it seems the development of the GGC has only recently come to fruition.
  
  
• Q: Why hasn't Google waited until June (e.g. for better pricing)?
• A: Competitive advantage.
• Come June, every Tom, Dick, Harry, Yahoo! and Microsoft will be presumably be buying large circuits to tie in local datacenters or extend peering.
• Moving first allows better publicity (even via word of mouth as opposed to above-the-line advertising) and can be used by their sales team as a competitive advantage.
  

Anything else?

AMS-IX breaks 500 Gbps!

[from the AMS-IX status page]

The number 1 internet exchange in the world by traffic volume - AMS-IX in Amsterdam - has reached 500 Gbps!



Notice the well-known flat summer trend. It's interesting to project what the max will reach by December: 550-600 Gbps?

Amazon's new CDN

Amazon's new CDN is cheered on by their CTO Werner Vogels. The primary take-away from the post:

Using a global network of edge locations this new service can deliver popular data stored in Amazon S3 to customers around the globe through local access.

Now, Amazon is only currently peering visibly with 4 other networks: Qwest, NTT, Tiscali, and Level3, which all seem to be transit providers.

Now I guess their edge routers don't absolutely need to sit in peered networks, but I'm sure it'll make the host networks happier [otherwise those host networks will be bearing the cost of reverse-proxying the traffic through their own transit connections].

The PeeringDB page for Amazon isn't that much more helpful - no public switching presence noted, but there is some promise shown in the following locations:

• Equinix Ashburn
• PAIX Palo Alto [incl. PacketExchange I guess]
• The Westin Building [Yay SIX! And PAIX Seattle too, I suppose]

Another development to watch closely. Amazon: where is your non-USA PoP presence? LINX and AMS-IX beckon...

TENET peers with Google

TENET's UK frontend Ubuntunet [wiki] is now peering with Google.

This is only in the UK for now, presumably they'll peer locally when Google gets their act together. Let's hope Google isn't waiting for the Seacom cable to land too.