# Events seperated by '---' # Each entry within an event should be date/time (YYYY-MM-DD HH:MM) on one line # with information on the following lines # # A date/time of: 0000-00-00 00:00 will be replaced with the current date/time # Useful for the "nothing wrong" entry, ie # # 0000-00-00 00:00 # We are currently unaware of any issues # # 0000-00-00 00:00 # We are currently unaware of any issues. Any issues should be reported via your normal support channels. --- 2018-01-19 Unexplained reboot of one server, loss of services hosted on thdo1. --- 2018-03-14 14:30 We've noticed connectivity issues to our servers, investigating. 2018-03-14 14:45 Issues creating new ARP entries, we are seeing sporadic connectivity at best. Waiting on resolution from ISP. 2018-03-14 15:30 An excess (many tens ot thousands of packets per second) of gleen traffic for an ubbound IP with another customer of the ISP is causing ARP issues. The IP in question has been null routed now. --- 2018-10-18 13:50 Network connectivity lost to storage sever. 2018-10-18 14:05 Resolved. Link flapping prevented failover and immediate restoration of connectivity. Resolved by ISP. --- 2019-01-21 16:00 We've noticed issues with email: '451 temporary local problem' 2019-01-21 16:20 Resolved. Intermittent issues with email due to problem with virus scanning. Now resolved and email is catching up. --- 2019-08-23 11:11 Service loss on thdo2 - investigating. 2019-08-23 11:20 Server appears to have failed, starting failover of services. 2019-08-23 12:15 Most services restored 2019-08-23 12:30 thdo2 restored --- 2019-11-21 22:37 Service loss on thdo2 - investigating. 2019-11-21 23:05 Failed over some database services 2019-11-21 23:10 Restarting services on thdo2 2019-11-21 23:40 Confirmed DDOS on against IP bound on thdo2 2019-11-21 23:45 Services fully restored --- 2020-09-27 23:20 Service loss on thdo1 - investigating. 2020-09-27 23:25 System locked, power cycling 2020-09-27 23:35 Restarting services on thdo1 2020-09-27 23:37 Services fully restored --- 2020-10-18 20:40 AT&T appear to have our mail server on a blocklist, we're unsure why. We have contacted them to request removal. This will impact email to att.net, bellsouth.net, flash.net, nvbell.net, pacbell.net, prodigy.net, sbcglobal.net, snet.net, swbell.net and wans.net 2020-10-22 02:20 AT&T resovled the accidental blacklisting --- 2020-12-15 22:00 We are seeing intermittent email issues to Google email hosted domains. Specifically 550-5.1.1 errors for known good addresses. 2020-12-15 23:00 It seems this is solely a Google end issue that is being investigated. Various 'down detectors' are showing Google email issues and some news stories are being logged, eg https://9to5google.com/2020/12/15/gmail-address-not-found/ They will presumably resolve the issue as quickly as possible. 2020-12-16 11:00 According to Google the issue is now resolved. https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8b67908fadee664c68c240ff9f529ab --- 2020-02-07 20:00 Loss of service on thdo5. Investigating prior to failing over services to other machines. 2020-02-07 20:30 All services restored after restarting hardware. --- 2021-03-03 07:10 Loss of service on legacy thdo1. Kernel panic. Restarting server. 2021-03-03 07:29 All services restored after restarting hardware. --- 2021-05-12 19:30 Loss of service on thdo2. Some loss of slave databases. 2021-05-12 20:30 Restoration of all services on thdo2 --- 2021-06-08 08:15 Dealing with DOS on thdo4, some loss of access to services 2021-06-08 09:09 Traffic to the target IP has been null routed. Other services should be available --- 2022-06-21 07:00 Cloudflare are experiencing issues this morning which is impacting a lot of sites (https://www.cloudflarestatus.com), our own services are all operating as expected. --- 2022-11-30 12:50 thdo3 and store0 have dropped offline. Investigating. 2022-11-30 13:26 Failed over some services. Telehouse are investigating the loss of connectivity. 2022-11-30 13:26 Vodaphone performed what they term a 'soft cease' on the rack with thdo3 and store0 in cutting network connectivity. We are unsure why. They have reversed this now. 2022-11-30 13:36 Vodaphone had been unable to determine who 'owned' the rack and rather than enquire of telehouse who had accessed it say in the last year they opted for the pull the fibres and see who shouted approach. Connectivity has been restored and the matter resolved. --- 2023-01-13 20:30 One of the power feeds to thdo1 and thdo4 has been taken out of service for maintenance by Telehouse until 1200 UTC on the 14th Jan. All equipment is functioning on the second power feed as expected. --- 2023-03-21 14:00 worldwidedns.net appear to be experiencing issues with services up and down and only partial name resolution. This is impacting access to some of our clients, our own services are all operating as expected. --- 2024-01-10 11:00 Our London location at Telehouse will be offline for some hours from 1100 UTC, services should be restored by 1600 UTC. All services other than those previously notified will be operating out of Portsmouth. 2024-01-10 11:38 Store0 going offline now for works. 2024-01-10 12:13 thdo3 going offline now for works. 2024-01-10 12:39 thdo4 going offline now for works. 2024-01-10 12:55 thdo1 going offline now for works. 2024-01-10 15:40 Store0, thdo1, thdo3 and thdo4 back online after works. --- 2024-12-03 01:05 Loss of one drive in mirrored pair on thdo3 2024-12-03 02:55 Loss of second drive in mirrored pair on thdo3, dataset is now degraded and hosted services on this node are offline 2024-12-03 08:00 Services on thdo3 migrated to thdo4. Investigation into disk failure underway. 2024-12-03 12:00 Disks showing no faults, pool has been brought back online. --- 2025-06-17 12:00 Routing from our Portsmouth data centre to London is currently taking the scenic route going about 500' between points in London via New York and Phoenix in the USA. We've raised the issue with the backbone carriers in question to resolve. Impact should be minimal, but you may experience slow loading times if resources are split between data centres. 2015-06-17 19:00 The 12,000 mile detour has been bypassed. An manual configuration on Telia's routers in Phoenix caused the "redirection" of traffic. We're back to a happier 3ms between data centres again. --- 2025-07-08 00:01 Expecting some loss of connectivity to Portsmouth DC as routers are upgraded. 2025-07-08 04:00 Portsmouth works completed with no loss of connectivity. --- 2025-07-30 22:46 Loss of connecticity to Portsmouth - investigating. 2025-07-30 22:55 Loss is partial and looks like a routing isssue. Datacentre have advised they are performaing maintenace and will investigate the connectivity issue. 2025-07-30 23:30 Bringing up services in London 2025-07-30 23:47 Network routing restored. 2025-07-31 00:40 Loss of connecticity to Portsmouth 2025-07-31 00:45 Bringing up services in London 2025-07-31 00:50 Network routing restored. 2025-07-31 01:00 Re-syncing services and starting failover of services to London 2025-07-31 02:22 DC advise the maintenace was to blame and was part of the router upgrade from 08-07-2025, but we had been missed off notification. Maintenance has been finished. --- 2025-08-16 09:00 We are currently experiencing difficulties sending email to Microsoft hotmail, outlook and live domains (365 hosted domains appear fine). This is due to a blacklist policy put in place by Microsoft. We're unsure what has caused the policy. We've raised the issue with Microsoft and requested de-listing and further information. 2025-08-16 22:00 It appears Microsoft have removed the blacklisting and email is once again being delivered to hotmail, outlook and live domains. --- 0000-00-00 00:00 We are currently unaware of any issues. Any issues should be reported via your normal support channels.