Google local peering starting (slowly)

Google now serves [almost] all traffic to TENET sites directly from either:

• peering connection, or
• some sort of GGC-like nodes [Google Maps, YouTube videos]
  

Name resolution and tracerouting from UCT hosts confirms that all Google services are being served from local [South African] servers. This has huge implications in bandwidth savings, and quality of user interaction because of lower latency for content that is already available on the local datacenter and caches.

Issues
Strangely, while www.youtube.com resolves to a locally-hosted server (64.233.179.100 currently), youtube.com (with no leading www.) resolves to a 208.65.153.xxx address, seemingly in Richmond, VA.

Clarification
The caching nodes [presumably GGC nodes] currently serve only certain services (map tiles, YouTube videos) and are generally co-located within a client network. The other Google services are served from machines located in Cape Town (based on ping times) and that are administratively within a Google-owned IP block.

Further clarification
Assume this is a "small" trial run (limited number of users on TENET, maximum goodwill from helping the poor academics) before a full roll-out. The caching nodes already save TENET up to 45 Mbps international transport! There is no local visibility to Google's ZA IP block from IS's route server.

TENET peering at LINX

More interesting traceroute data [target www.yahoo.com]:

8 v2750-tiger-brie.uni.net.za (155.232.145.226) 2.732 ms
9 unknown.uni.net.za (196.32.209.25) 152.780 ms
10 ge-1-1-0.pat1.the.yahoo.com (195.66.224.129) 154.808 ms
11 so-0-1-0.msr1.ird.yahoo.com (66.196.65.33) 163.932 ms
12 gi-1-1.bas-b1.ird.yahoo.com (87.248.101.1) 164.020 ms
13 f1.us.www.vip.ird.yahoo.com (87.248.113.14) [open] 165.543 ms


The TENET router in London (196.32.209.25) connects to the Yahoo! AS, and the 195.66.224.0/19 router address is a giveaway that it's through LINX.

Testing other networks [target www.facebook.com] yields more data:

9 unknown.uni.net.za (196.32.209.25) 153.603 ms
10 linx.br01.lon1.tfbnw.net (195.66.225.69) 153.336 ms
11 xe-x-x-x.br01.ash1.tfbnw.net (204.15.22.245) 229.906 ms
12 te-13-0.csw06a.ash1.tfbnw.net (204.15.23.55) 233.942 ms
13 www.11.06.ash1.facebook.com (69.63.186.12) [open] 262.342 ms

where hop 10's hostname confirms the LINX connection, and also shows we're still connecting to the Virginia servers.

The LINX looking glass also shows the routing to TENET through AS1299 (Telia) and AS2914 (NTT) and directly.

Traffic graphs confirm significant traffic through LINX


Several other autonomous systems are reachable via LINX peering sessions, including hosting providers Hurricane Electric and LeaseWeb and CDN Limelight Networks [ex YouTube providers, and 2009 Presidential inauguration streamers].

Clearly this is in preparation for the new SEACOM pipe, although TENET is saving on transit requirements.

Google 'local peering' live?

From some traceroute data [target: www.blogger.com]:

9 unknown.uni.net.za (155.232.253.254) 2.341 ms 2.055 ms 2.237 ms
10 64.233.174.40 151.205 ms 151.276 ms 151.457 ms

Note the jump in RTT as we go from the TENET router to the Google router (64.233.160.0/19 belongs to Google).

Compare to traceroute data for a less aggressively-peering company [target: www.microsoft.com]:

8 v2750-tiger-brie.uni.net.za (155.232.145.226) 2.341 ms 2.254 ms 2.337 ms

9 unknown.uni.net.za (196.32.209.25) 153.050 ms 152.938 ms 152.223 ms

Where the next hop is a TENET-controlled router overseas.

So perhaps Google has started 'peering locally', and are transparently tunneling the data overseas to allow IS [and/or other ISPs] to avoid paying for each user's international bandwidth. Of course Google are therefore paying for it, but with the standard international bandwidth pricing feedback damping the usage, they probably aren't seeing a huge explosion in bandwidth demand.

Contrast this to the local YouTube traffic caching, where having locally-available video means that videos load faster: better user experience translates to higher usage, and the demand increases. For the user this is a virtuous circle, but to the various I[C]T[S] departments it must seem more like a vicious circle.

However, for organisations like TENET where the pipe is purchased old-school (i.e. 'per Mbps' rather than 'per GB'), this will be a boon, because up to 40 Mbps [peak] is now being handed-off locally. Interestingly the upstream/outbound traffic, presumably web crawling, is still going through the Ubuntunet point in London.

Some graphs:

The London peering session going [relatively] quiet late Tuesday.

A 'local peering' session [presumably with Google] in Cape Town going live late Tuesday:

And some of the local caching traffic seems strongly correlated (via Mk. 1 eyeball-based peak matching on other graphs) to the local peering session.

The takeaway: Google is probably peering locally, but is trying to avoid flooding ISPs with bandwidth-conversion economic problems.

iPhone + Geolocation + P2P

Since Apple's announced new embeddable Google Maps for the iPhone, not to mention the new P2P support, perhaps it's not too much to ask to get a cool new proximity-based alerting system?

Such a system could be used for geocaching or even friend-finding (the real kind of friend-finding where you already know the person...)

TENET has new transit providers

The TENET traffic graphs show NTT as a new provider:

And a 10G interface, to boot! This is likely in preparation for the arrival of SEACOM, as TENET's international bandwidth is the current bottleneck.

Traffic on Datahop has been halved, and traffic on Telias is at a quarter to a third of normal rates. So they are either being phased out or being used as backup providers.