[IPv6crawler-wg] Note to Clients and Colleagues / GIH email disruption

[Olivier MJ Crepin-Leblond]

15 February 2010

Dear Clients and Colleagues,


I wanted to write a quick note to explain the recent serious disruption
we have had with our telecom systems and how it has hindered our business.

On Thursday 11 February 2010 at around 10:55am, the GIH office lost
connectivity with the outside world, and that affected all of our
servers on-site which were, effectively cut-out from the rest of the world.
Upon testing the line remotely, British Telecom (BT) engineers found
that there was a problem at the BT exchange, and as we pay a premium to
be one of their priority customers, they would have this link again
within 24h.

Due to "an extremely busy day", BT was unable to dispatch an engineer
on-site on Friday, but promised to fix the problem by the next day.

On Saturday afternoon, a BT engineer came to the GIH office and tested
the line from our side, only to say "there is something seriously wrong
- it makes a sound like a fire engine" and proceeded to the South
Kensington Exchange to fix the fault. He called us from there to tell us
that the exchange was completely deserted and that he could not fix this
by himself so the work was moved to Monday - this morning.

On Monday morning, after spending 45 minutes with the operator, I was
told that an engineer was in the South Kensington Exchange, and an
engineer onsite and they were carrying out tests on the line. I was also
told that our second telephone line into the office is also facing a
fault, although it appears that it is a different fault than the first
one - much like the fault in the office next to ours. So no back-up link
possible. Two hours later, I spent another 20 minutes with BT faults,
and they announced to me that this is a major fault, with their cable
being damaged and no spare capacity at all in the area, which requires
closing the road, installing alternative traffic lights, digging the
road up and working on the cables directly. Work would start on
Wednesday 17th February due to the potential for big disruption of
London traffic.

I therefore arranged for our servers and router to be physically moved
from the GIH office to our providers, the Twenty Twenty Media offices in
South London. They are now in place in on a temporary basis, and are
running smoothly again.

Customers and colleagues routing their email through salsa.gih.co.uk can
use the service again since it is stable.

A backlog of 59 000 emails currently being stored at our back-up MX,
psg.com, is being processed but will take a while to clear, so if you
have not received a reply from us, please accept my most sincere apologies.
These exceptional circumstances were completely outside our control.

Special thanks go to Zsena Ka for coming in the office on Saturday and
Rex Wickham (2020Media) for doing an exceptional job on Monday to move
the servers single-handedly, in spite of traffic problems and the
inability to park in the middle of the day in the busy High Street
Kensington Area.

Your sincerely,

-- 
Olivier MJ Crépin-Leblond, PhD
http://www.gih.com/ocl.html