Matrix

365 readers
1 users here now

The secure messaging protocol.

founded 1 year ago
MODERATORS
51
52
1
submitted 7 months ago* (last edited 7 months ago) by kate@lemmy.uhhoh.com to c/matrix@programming.dev
 
 

minor bug fixes but good version number

53
54
 
 
55
56
 
 

The Matrix.org homeserver ^[2]^ is currently experiencing a "major outage" ^[1.1]^ — the server is currently offline ^[1.3]^. The timeline of events is currently (2025-09-03T00:14Z) as follows:

  • On 2025-09-02T17:39Z, an issue with the matrix.org database was reported by status.matrix.org ^[1.3]^.
  • At 2025-09-02T19:01:27Z, the Matrix.org Foundation later reported that the matrix.org secondary database lost its filesystem at 2025-09-02T11:17Z, and subsequently lost its primary database filesystem at 2025-09-02T17:26Z. ^[5]^
  • At 2025-09-02T21:39:25Z, the Matrix.org Foundation reported that they were unable to restore the primary database filesystem, so they are restoring a 55TB database snapshot from the previous night. This restoration is expected to take more than 10 hours to recover the data, then greater than 4 hours to restore the data, then greater than 3 hours to catch up on missing traffic. ^[6]^

References

  1. Type: Website. Publisher: "Matrix.org". Accessed: 2025-09-03T00:18Z URI: https://status.matrix.org/.
    • This is the "system status and incidents" website for The Matrix.org Homeserver ^[4]^
    1. Type: Image.

    2. Type: Image.

    3. Type: Image.

  2. Type: Meta.
    • As seen on status.matrix.org, the site which tracks the system status and incidents of The Matrix.org Homeserver ^[3.1]^, the section "Matrix.org" contains the context: "The Matrix deployment on matrix.org" ^[1.2]^. Within that section, Synapse, which is reported to have the outage ^[1.1]^ is a Matrix homeserver ^[5]^.
  3. Type: Article. Title: "The Matrix.org Homeserver". Publisher: "The Matrix.org Foundation". Accessed: 2025-09-03T00:30. URI: https://matrix.org/homeserver/about/.
    1. Type: Text. Location: ¶3.

      System status and incidents

  4. Type: Website. Title: "Servers". Publisher: "The Matrix.org Foundation". Accessed: 2025-09-03T00:38Z. URI: https://matrix.org/ecosystem/servers/.
    • Type: Image.

  5. Type: Post>Text. Author: "The Matrix.org Foundation" ("@matrix@mastodon.matrix.org"). Publisher: "mastodon.social". Published: 2025-09-02T19:01:27.000Z. Accessed: 2025-09-03T00:46Z. URI: https://mastodon.matrix.org/@matrix/115136245785561439.

    So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.

    • "FS" is presumed to mean "filesystem" from

      […] the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS […] ^[5]^

      combined with

      […] we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary […] ^[6]^

    • "earlier today (11:17 UTC)" is assumed to mean 2025-09-02T19:01:27Z as this post was published on 2025-09-02T19:01:27.000Z.

    • Since no other qualifying information is given, and given its successive nature to 11:17, "17:26" is presumed to mean 2025-09-02T17:26Z.

    • […] Then, we lost the primary at 17:26 […]

      This is presumed to be referring to the primary database filesystem given the context:

      […] the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26 […]

    • "DB" is presumed to mean "database" given the following context

      […] the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS […]

  6. Type: Post>Text. Author "The Matrix.org Foundation" ("@matrix@mastodon.matrix.org"). Publisher: "mastodon.social". Published: 2025-09-02T21:39:25.000Z. Accessed: 2025-09-03T00:50Z. URI: https://mastodon.matrix.org/@matrix/115136866878237078.

    Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic. Huge apologies for the outage. Again, folks using their own homeservers are not impacted.

    • "DB" is presumed to mean "database" ^[5]^.
57
 
 
58
 
 
59
60
61
62
63
 
 

Last month we issued a Pre-disclosure: Upcoming coordinated security fix for all Matrix server implementations, describing a coordinated release to fix two high severity protocol vulnerabilities (CVE-2025-49090; the other not yet allocated a CVE). That release is now available as of 17:00 UTC on August 11, 2025. Server updates are now available, and MSCs & spec updates will follow on Thursday, August 14, 2025, bringing us to version 1.16 of the spec later in the month, and introducing room version 12.

If your rooms or spaces federate with untrusted servers, you should plan to upgrade your rooms to room version 12. The urgency of this upgrade may depend on your community’s readiness for the changes. At the Foundation, we are aiming to upgrade our rooms in September 2025. There needs to be enough time to allow clients and servers participating in your room to support v12 before upgrading your room.

The new version includes some changes to room creator semantics, which means that choosing which user performs the upgrade needs some careful thought. Using a long-lived, trusted account, such as a moderation bot account, is advised. For more detailed advice, two of the Foundation Governing Board working groups — the Trust & Safety Research & Development Working Group, and the Website & Content Working Group — have collaborated on a guide for upgrading rooms and spaces to version 12. That guide will help you to plan your upgrades and to make them happen.

They still aren't releasing details of the bug yet, but reading between the lines it clearly allows malicious servers to do things they shouldn't be able to. presumably it will become clearer on thursday when they publish the new specs which monday's software releases implement.

64
 
 

Pair a good matrix instance with a good client and you're golden.

65
66
67
 
 

October 15 to 18, 2025
Strasbourg, France

68
69
70
 
 

https://matrix.org/homeserver/pricing/

Usage limits may change over time as we refine the offering.

71
72
73
74
75
view more: ‹ prev next ›