IT disruption ‘unacceptable’ and ‘should not happen’

public services has assured islanders.

A faulty air conditioning unit in the equipment room at Frossard House was the first issue which led to the States’ systems being offline for several days recently.

This was followed by the failure of the backup AC unit, causing the hundreds of servers in the room to overheat.

A further technical problem saw backup systems at Edward T Wheadon House failing to take over.

The causes are still being investigated, said head of the public service Mark de Garis, but once established a report would be prepared.

The equipment room houses about 500 servers which contain data and applications for different States departments.

All of the software and data is backed up regularly to hardware at Edward T Wheadon House, with the idea being if there was a major failure at Frossard House, the parallel system at Wheadon House could take over.

This did not happen on this occasion, however, and is one of the key issues being investigated.

All of the data is backed up to a third site, too, and none was lost, said Mr de Garis, who repeated his apology for what happened.

‘I really want to apologise again to everybody that’s been affected by this because this shouldn’t happen,’ he said. ‘It is unacceptable.’

An alarm was triggered when the temperature in the equipment room hit 25C early in the morning of 25 November. When the

servers got too hot, they shut down to protect themselves and the data.

Technicians were on site in 20 minutes and had to call in a third party contractor to provide cooling for the room, which ended up reaching 48C.

Mr de Garis said the applications which use the data were still having problems as work is undertaken to restore the services.

He was unable to confirm when the systems were last tested before the incident.

The States’ partnership with Agilisys has led to a lot of old applications being replaced and will eventually see all of its servers housed in two local data centres.

‘We have over 200 computer systems and 89 of those have already been taken into new data centres... and those services were completely unaffected by this outage,’ said Mr de Garis.

But it is a ‘huge, huge project’, he said. Once completed, systems will not have the same degree of vulnerability.

He said the eventual report would have some internal security information which would have to remain private, but the findings and any recommendations would be made public.

He was unable to estimate how long it might be before the report was completed, although work had already started.

‘As soon as we’ve got the resilience to the system back up and corrected everything in Frossard House we’ll be putting all efforts into that [report].’