Ramping up Research Traffic on SEACOM

While TENET has enjoyed high-speed interchange of traffic with GÉANT before, it has mostly been from its nodes co-located in London (e.g. local mirrors).

However, it looks like things are heating up - traffic of up to 700 Mbps between the SANReN ring in Gauteng...

and GÉANT in Europe...

Interestingly, none of the TENET graphs show which client networks this is going to, so perhaps this is just an internal test or mirror population?

Return of CINX

The ISPA has re-launched CINX (the Cape Town Internet Exchange). PR here.

Nothing on the old CINX and why it was discontinued, except for a rather oblique statement that there is now has "more than enough Internet traffic to justify an exchange for the city".

No graphs for the exchange yet, but the TENET folks are on the ball and you can view their port graphs here. 10 Mbps in Cape Town doesn't seem like much compared to the 100 Mbps of JINX (which looks suspiciously flat, although there is a 175 Mbps spike in there).

Excellent news for local providers.

End of an era

TENET now runs all institutional traffic over SEACOM:

Bandwidth usage on the SAT-3 portions has dwindled to almost nothing; the SAT-3 connectivity should terminate sometime in the next month or so.

Interesting to see some BGP traffic on the Telia backup link - perhaps routing changes to some of the institutional ASs propagating through and settling down?

Google preparing for full roll-out?

Google shuffles around some more DNS records et voilá:

• mail.google.com now resolves to a Google-owned IP (one of the cpt01s01 series)
• mt.google.com now resolves to a Google-owned IP (as above, so below)

Was the hosting of the Google Maps tiles on the [presumed] GGC caching box just a fig-leaf, a ploy for respectability? Since only YouTube access still seems to be directed towards those caching boxen, perhaps that traffic was a quite-substantial chunk of the original Google traffic to TENET's network, the rest of which is now easily handled by the local peering arrangement.

It might allow easier filtering or shaping for the IT/ICTS departments so inclined, but why else keep the YouTube traffic on those boxes? Google's purported 'boxenertia'?

However, seeing that GMail is now locally hosted, that means the major services are being served from local machines. Perhaps this means Google is getting ready for a full-bore load test, in preparation for a proper roll-out?

Google tuning

Even though Google's new Cape Town nodes are in limited production / test mode, users have been connecting to both the local data-centre, as well as to overseas datacentres [e.g. London?].

Presumably Google is still tuning the DNS based on usage while simultaneously expanding the datacentre capacity.

Interestingly it is only mail.google.com that returns a non-local host. So apparently mail is not amenable to normal caching modes or it would have been done through either the [presumed] GGCs or even transparently cached within the Cape Town datacentre.

Let's hope that Google get some cheap capacity on SEACOM even though is has higher latency than SAT-3 (~215 ms vs ~150 ms). Since most of the front-end logic (e.g. client javascript) would be served by local machines, interactivity should not be compromised too much.

Mmmm mirrors

Wow.

1.2+ Gbps traffic from the mirror.ac.za front-end in London.

Google local peering starting (slowly)

Google now serves [almost] all traffic to TENET sites directly from either:

• peering connection, or
• some sort of GGC-like nodes [Google Maps, YouTube videos]
  

Name resolution and tracerouting from UCT hosts confirms that all Google services are being served from local [South African] servers. This has huge implications in bandwidth savings, and quality of user interaction because of lower latency for content that is already available on the local datacenter and caches.

Issues
Strangely, while www.youtube.com resolves to a locally-hosted server (64.233.179.100 currently), youtube.com (with no leading www.) resolves to a 208.65.153.xxx address, seemingly in Richmond, VA.

Clarification
The caching nodes [presumably GGC nodes] currently serve only certain services (map tiles, YouTube videos) and are generally co-located within a client network. The other Google services are served from machines located in Cape Town (based on ping times) and that are administratively within a Google-owned IP block.

Further clarification
Assume this is a "small" trial run (limited number of users on TENET, maximum goodwill from helping the poor academics) before a full roll-out. The caching nodes already save TENET up to 45 Mbps international transport! There is no local visibility to Google's ZA IP block from IS's route server.

TENET peering at LINX

More interesting traceroute data [target www.yahoo.com]:

8 v2750-tiger-brie.uni.net.za (155.232.145.226) 2.732 ms
9 unknown.uni.net.za (196.32.209.25) 152.780 ms
10 ge-1-1-0.pat1.the.yahoo.com (195.66.224.129) 154.808 ms
11 so-0-1-0.msr1.ird.yahoo.com (66.196.65.33) 163.932 ms
12 gi-1-1.bas-b1.ird.yahoo.com (87.248.101.1) 164.020 ms
13 f1.us.www.vip.ird.yahoo.com (87.248.113.14) [open] 165.543 ms


The TENET router in London (196.32.209.25) connects to the Yahoo! AS, and the 195.66.224.0/19 router address is a giveaway that it's through LINX.

Testing other networks [target www.facebook.com] yields more data:

9 unknown.uni.net.za (196.32.209.25) 153.603 ms
10 linx.br01.lon1.tfbnw.net (195.66.225.69) 153.336 ms
11 xe-x-x-x.br01.ash1.tfbnw.net (204.15.22.245) 229.906 ms
12 te-13-0.csw06a.ash1.tfbnw.net (204.15.23.55) 233.942 ms
13 www.11.06.ash1.facebook.com (69.63.186.12) [open] 262.342 ms

where hop 10's hostname confirms the LINX connection, and also shows we're still connecting to the Virginia servers.

The LINX looking glass also shows the routing to TENET through AS1299 (Telia) and AS2914 (NTT) and directly.

Traffic graphs confirm significant traffic through LINX


Several other autonomous systems are reachable via LINX peering sessions, including hosting providers Hurricane Electric and LeaseWeb and CDN Limelight Networks [ex YouTube providers, and 2009 Presidential inauguration streamers].

Clearly this is in preparation for the new SEACOM pipe, although TENET is saving on transit requirements.

Google 'local peering' live?

From some traceroute data [target: www.blogger.com]:

9 unknown.uni.net.za (155.232.253.254) 2.341 ms 2.055 ms 2.237 ms
10 64.233.174.40 151.205 ms 151.276 ms 151.457 ms

Note the jump in RTT as we go from the TENET router to the Google router (64.233.160.0/19 belongs to Google).

Compare to traceroute data for a less aggressively-peering company [target: www.microsoft.com]:

8 v2750-tiger-brie.uni.net.za (155.232.145.226) 2.341 ms 2.254 ms 2.337 ms

9 unknown.uni.net.za (196.32.209.25) 153.050 ms 152.938 ms 152.223 ms

Where the next hop is a TENET-controlled router overseas.

So perhaps Google has started 'peering locally', and are transparently tunneling the data overseas to allow IS [and/or other ISPs] to avoid paying for each user's international bandwidth. Of course Google are therefore paying for it, but with the standard international bandwidth pricing feedback damping the usage, they probably aren't seeing a huge explosion in bandwidth demand.

Contrast this to the local YouTube traffic caching, where having locally-available video means that videos load faster: better user experience translates to higher usage, and the demand increases. For the user this is a virtuous circle, but to the various I[C]T[S] departments it must seem more like a vicious circle.

However, for organisations like TENET where the pipe is purchased old-school (i.e. 'per Mbps' rather than 'per GB'), this will be a boon, because up to 40 Mbps [peak] is now being handed-off locally. Interestingly the upstream/outbound traffic, presumably web crawling, is still going through the Ubuntunet point in London.

Some graphs:

The London peering session going [relatively] quiet late Tuesday.

A 'local peering' session [presumably with Google] in Cape Town going live late Tuesday:

And some of the local caching traffic seems strongly correlated (via Mk. 1 eyeball-based peak matching on other graphs) to the local peering session.

The takeaway: Google is probably peering locally, but is trying to avoid flooding ISPs with bandwidth-conversion economic problems.

TENET has new transit providers

The TENET traffic graphs show NTT as a new provider:

And a 10G interface, to boot! This is likely in preparation for the arrival of SEACOM, as TENET's international bandwidth is the current bottleneck.

Traffic on Datahop has been halved, and traffic on Telias is at a quarter to a third of normal rates. So they are either being phased out or being used as backup providers.

UCT permanent upgrade to 43 Mbps

The bandwidth test was obviously successful as UCT has upgraded their connection to 43 Mbps. Graphs at the usual place [notice the jump around 12.30pm].

ICTS has issued a remarkably restrained notice. Did they not trumpet this to the whole campus because:

• it's a drop in the bucket compared to the demand?
• they don't want to stimulate further demand?
• they don't want any recognition?

Whatever the reason, we still look forward to the upcoming 1 Gbps temporary link for TSN 63 (UCT)

Google resolving locally

When resolving www.google.co.za from within TENET IP space, one now receives A records indicating TENET IP addresses.

There are still some holdouts - kh.google.com / mwx.google.com and mail.google.com still resolve to normal Google IP addresses. Perhaps they'll transition soon.

Wits trounces UCT

I'm not writing about sports now, but rather about Wits's recent climb to the number 1 position on the unofficial bandwidth rankings:

Their monitoring graph shows peaks above 60 Mbps, and TENET's order status page shows an upgrade of their backbone link to 57920 kbps.

UCT's monitoring page show the same old peaks, with an upcoming 43008 kbps link as a consolation prize. This is obviously what was under test over the last week or two.

And it's interesting to see a new 'five-thirds' formula mentioned. Taking UCT's international commit of 26 Mbps and multiplying by 5/3 yields about 43 Mbps. Fully 40% of traffic is expected to be local! TENET: how about showing some flow statistics to the IS Akamai clusters?

UCT's extra bandwidth redux

[from a comment on the last post:]

Of course, SANREN is what's supposed to provide the drastically improved last-mile.

SANReN will eventually provide the core of the new network, and maybe the access leg too. Some Gauteng institutions have access via the fibre ring there and are happy. Many large institutions elsewhere don't have that yet (and are waiting, waiting, waiting) and are unhappy.

However, if the UCT link was increased from ~32 to ~44 Mbps (coincidentally this is like going from an E3 to a DS-3), what's to prevent it from being increased further? Why not a DS-4 (~274 Mbps) :)

BTW, the Google caches are already live. [We don't know for sure that these are GGCs, but see below ] Their effectiveness seems to be improving (presumably their caches are still filling), but they do still use quite a lot of incoming (international) bandwidth.

If these are Google caches (GGCs), I doubt they will have reduced international bandwidth usage much. The local usage would spike because access is so much quicker and easier.

Traceroute (ICMP, rather than UDP) to v1.cache.googlevideo.com

mt.google.com is also mapping to a TENET IP block, so Google Maps tiles should be served from somewhere local [TENET or GGC or who-knows-what].

kh.google.com [Google Earth] is still mapping to the Google IP addresses overseas.

mail.google.com and docs.google.com still map to the Google IP blocks overseas.

So either TENET has some sort of fancy caching and line-rate traffic redirection going on, or Google is doing something interesting.

Hopefully the local build-out will be finished soon.

UCT's extra bandwidth

UCT seems to have gotten some extra temporary bandwidth today. Looking at the graphs we see a spike at 14h00 (GMT+2) in the traffic graphs.



Most of the increased bandwidth usage seems to have consumed international traffic and so that international portion of UCT's bandwidth allocation is finally pegged against the limit of 26 Mbps.



Which is interesting.

Because it means that until now, the international traffic has been crowded out by national traffic. Much of which will be to the Akamai cluster at IS, and the upcoming Google Global Caches (GGCs) and local servers.

So more international bandwidth isn't going to help unless the actual "last-mile" to UCT is drastically improved.

TENET peers with Google

TENET's UK frontend Ubuntunet [wiki] is now peering with Google.

This is only in the UK for now, presumably they'll peer locally when Google gets their act together. Let's hope Google isn't waiting for the Seacom cable to land too.

Facebook CDN

I don't know how this slipped through the cracks, but I've just noticed that Facebook has started hosting some content on static.ak.fbcdn.net [fbcdn presumably = facebook content distribution network]

Is Facebook planning on using multiple CDNs?

Both static.ak.fbcdn.net and static.ak.facebook.com seem to be hosted by Akamai, so for us "lucky" .ac.za we get routed through IS. This is actually good news, as we have a very high-speed connection to IS from TENET. Unfortunately, most institutions connect slowly into TENET (~30 Mbps for UCT!) and implement some shaping on popular websites. So we don't get most of the benefit.

Extra transit for TENET

Viewing the monitoring graphs for TENET's London node, a new transit circuit through DataHop [web?]is apparent:


Peak speeds are about 50 Mbps. Combined with the Telia peak of ~100 Mbps, this means TENET currently has ~150 Mbps peak transit.

Note the very small outbound. Mmmm, eyeballs.

Unscheduled scheduled outages

A recent posting to the TENET ren-news list:

All Neotel MPLS GEN3 sites experienced a number of service failures
early yesterday morning. It appears that these were caused by Neotel
maintenance, of which TENET was not informed in advance.

even given

The Service Level Agreement provides that: "...The Providers will give TENET four days' advance notice of planned maintenance."

Let's hope Neotel doesn't become Neotelkom [the new Telkom?] :)

ICTS vs users

For some reason there is an article praising UCT ICTS's War on Terror...er...Bandwidth:

[snip lots]
ICTS has implemented Packeteer Packetshaper
[snip lots more]

AFAIK this actually happened ages ago. So there's no real reason to trot this out now unless people are griping that GEN3 isn't doing much for them.

And there's no mention of what will happen once:

• UCT transitions to SANReN and has a 10 Gbps link to the local node, and
• TENET moves to the 10 Gbps overseas pipe due mid-2009

I don't see the Packeteer appliance scaling to multi-Gbps speeds without serious extra $$$ invested. And would it be worth doing so?

Consider this paper on web search clickstreams http://doi.acm.org/10.1145/1177080.1177110]:

The MWN [Munich Scientific Network] provides a 10Gbit/s singly-homed Internet connection to roughly 50,000 hosts at two major universities along with additional institutes; link utilization is typically only around 200–500 Mbit/s in each direction.

Now this might have been back in 2006; it doesn't really matter if the traffic demand has doubled since then. Also, since major local and international peering will then be in effect, actual transit pressure should decrease.

C'mon ICTS - plan ahead now!