All systems are operational

Past Incidents

22nd February 2020

No incidents reported

21st February 2020

No incidents reported

20th February 2020

No incidents reported

19th February 2020

No incidents reported

18th February 2020

No incidents reported

17th February 2020

Compute Nodes Outage

We have lost connectivity to our Sydney location and are currently awaiting a response from our provider.

13:21 GMT: Expected resolution time is within 30 minutes.

13:25 GMT: Services are back online.

16th February 2020

CreeperPanel Slowness

We've noted slowness when logging into CreeperPanel, this is likely due to high load from repeated login attempts.

Compute Nodes Networking Event

A networking event (as yet unidentified) has caused customer containers to be unable to connect outbound.

We are investigating this and applying fixes.

19:23 GMT: we've identified a top of rack switch which has rebooted unexpectedly. This caused a drop in connectivity to host nodes and upon restoration, it appears the containers network interfaces did not come back up, as such we are going through and manually restart the containers. We will update this further when we can.

19:58 GMT: All containers have been restarted, any further issues please open a ticket to our support team.

15th February 2020

No incidents reported

14th February 2020

No incidents reported

13th February 2020

Buffalo Thermals, scheduled 1 week ago

We are installing additional cooling to many of our new Buffalo nodes, this will result in rolling reboots.

12th February 2020

No incidents reported

11th February 2020

No incidents reported

10th February 2020

No incidents reported

9th February 2020

No incidents reported

8th February 2020

No incidents reported

7th February 2020

No incidents reported

6th February 2020

Compute Nodes lax110 node issues

We identified an issue regarding one of our nodes where vms did not have network connectivity. Upon rebooting the node to attempt to resolve this, it did not come up correctly.

We are liaising with the remote data centre and will update when we have further information.

Update: 2020-02-06 17:54 UTC

This has now been resolved.

5th February 2020

No incidents reported

4th February 2020

No incidents reported

3rd February 2020

No incidents reported

2nd February 2020

No incidents reported

1st February 2020

No incidents reported

31st January 2020

Los Angeles-1 API Endpoint AS Loop in communications between Miami and Los Angeles

We have taken our Los Angeles API endpoint down as there is currently a network routing loop causing communications on the return path from some of our locations (namely, Miami) to fail, meaning the API will wrongfully report services as offline and be unable to control them.

Traffic will be routed automatically to other API endpoints, but this may result in slower control panel performance than usual.

We've worked with our transit provider to exclude our routes from the offending upstream.

30th January 2020

No incidents reported

29th January 2020

No incidents reported

28th January 2020

Compute Nodes Buf152 outage

Buffalo node #152 was detected as frozen.

6:30 am GMT; Upon reboot it is displaying memory errors. 6:52 am GMT; Remote hands have been requested to reseat and if still failing, replace the faulty memory module(s). 7:29 am GMT; Reseating the memory has allowed the machine to boot. 8:10 am GMT; We've taken the machine down to fully test the memory to ensure this does not recur. 8:47 am GMT; Passed first test, bringing machine back online and will monitor.

27th January 2020

No incidents reported

26th January 2020

No incidents reported