|
|
2024-10-14 0900h: I am aware of no service-affecting issues at this time. |
2024-10-12 1745h: It's clear that the system has power, and the network is certified fine by the colo guys. I suspect the kernel hit some edge case and panicked, but I won't know for sure until I go by the colo tomorrow. News as it's made.
2024-10-12 1559h: teaparty.net appears to be completely unreachable. I do not yet know why.
2024-03-22 0830h: Everything is down while I reinstall teaparty, yet again.
2024-03-21 0840h: All user-facing services except the photo gallery and webmail are back, but for performance reasons we're going to have to do this all again; we will be down all of Fri 22/3 while I re-reinstall.
2024-03-13 1300h: All user-facing services except mailman (the mailing list manager), webmail, and the photo gallery are now back and working. The system is running slowly, and a hardware upgrade has been put in progress to remediate that over the next 2-3 weeks. We have definitively established that mailman3 can't be made to run on teaparty without significant upheaval. I'm starting to try to set it up elsewhere, and migrate the lists. Webmail is also an unsolved problem, and may not come back. The photo gallery should eventually return, but it'll be a while.
2024-03-12 1750h: I think I have a handle on the performance issue, and it's now under control, though still not great. Teaparty will be slower than you're used to, but mail in and out should now be working with the exception of the mailing lists.
2024-03-12 1455h: A serious performance issue has been discovered with the new configuration. It is sufficiently bad that teaparty is borderline unusable. I am trying to come up with an optimum plan to proceed from here, but it must be expected that teaparty will be down for some days.
2024-03-11 0645h: Teaparty is expected to be substantially or completely down 11-13/3 for a complete reinstall and OS upgrade.
2023-11-02 0937h: teaparty is currently unreachable from my monitoring server via ipv6 only; ipv4 access remains fine. I also have several other test endpoints that can't reach teaparty via ipv6. A call is open with my colocation provider, and we're trying to get to the bottom of it. Meanwhile, if you experience any problems with ipv6 access, I recommend falling back to ipv4.
2022-02-03 1000h: we expect teaparty to be more down than up until 1100h, while it is moved to a different rack in the colo.
2020-12-12 1130h: we're currently back on-air, but I'm told that only a single fibre strand has been reconnected, and we are therefore still very much at-risk of both intermittent service and sudden repeated outage.
2020-12-12 1025h: current best time-to-fix is midday.
2020-12-12 1010h: emergency fibre maintenance works near the colo, which should have affected only one of the dual paths in, have affected both. Nobody is quite sure why, yet. There is no estimated time to fix.
2020-12-12 0940h: the issue continues. The backhaul provider, Vorboss, are confirming the loss of multiple links, so at the moment it's looking like someone put a JCB through a conduit close to the A1(M), or equivalent disaster.
2020-12-12 0848h: the colo confirms a routing issue.
2020-12-12 0840h: there is an issue accessing teaparty.net. I have no confirmation from the colo yet, but the loss of access to other equipment in an adjacent netblock suggests it's a network failure rather than teaparty itself. News as it's made.
2020-10-19 0905h: IPv6 is currently not routing correctly to or from teaparty.net. IPv4 service is unaffected. A service issue has been logged with the hosting company and their response is awaited.
2020-10-13 1045h: teaparty went through a period of poor availability, but things seem to be better now. I'll be holding a watching brief for the rest of the morning.
2020-10-13 0934h: the data centre that hosts teaparty.net is experiencing network instability. See their status page for up-to-date information. At the moment this doesn't appear to be impacting signficantly on teaparty's performance.
2020-07-31 1030h: I think we're back in business. I'll be at the colo for another 30-60 minutes, CALL ME ON MY MOBILE if you have issues.
2020-07-31 0745h: the restore seems to have gone fairly well overnight (FAOD: all user data is safe). There are bugs, and I'm working through them, but no show-stoppers as yet. At the moment I'm intending to take teaparty back to the colo sometime this morning, which (in an ideal world) would mean it was up and running by noon. This is not a promise and could change at any minute.
It turns out that we were just early discoverers of a massive thumping bug that's trashing UEFI systems left, right, and centre. I'm not sure if that makes me feel better or not.
2020-07-30 1655h: The issue seems to have been related to bad UEFI implementation in the hardware, which turned a bog-standard kernel upgrade into a brick-the-system episode. I am reinstalling the system without UEFI support. This will take some hours, and even when done, I intend to watch the system like a hawk before taking it back to the colo. If all goes very well: late tomorrow. If all doesn't: could be days. Possibly a week.
2020-07-30 1205h: System data (stored emails, etc.) seem to be fine, but the system won't boot, and resists attempts to make it do so. I am taking it home, because it's easier to work on there. Service is very unlikely to be restored today. Tomorrow is possible but don't in any way shape or form rely on it.
2020-07-30 1055h: teaparty.net is completely down. Tom is onsite at the colo investigating. There is no time-to-fix yet.