Original text below this.

Following release of lemmy 0.19.18 we've refederated as this release should fix the bug that caused the issue.

TL;DR

We’ve temporarily defederated from Hexbear due to a Lemmy bug with very deeply nested comment threads.

A thread there triggered repeated crashes on our server, causing errors like 502 pages and “Lemmy is starting” messages. Defederating stops the issue for now.

Announcement

Due to technical issues, we’ve temporarily defederated from Hexbear until a Lemmy update is available that fixes issues with deeply nested comment chains.

There is a known bug in Lemmy (see: https://github.com/LemmyNet/lemmy/issues/6435 ) where very deeply nested comments can trigger excessive recursion during federation. When Lemmy processes these comments, it recursively fetches and verifies parent comments, which can eventually lead to stack overflows.

Under normal circumstances this happens rarely (we’ve been seeing it maybe once per day), but it becomes much more problematic when multiple new comments are added to an already deeply nested thread. Each new activity can trigger processing of the same deep chain again.

In this case, a thread on Hexbear received a large number of additional replies in a very deep comment chain.

This caused Lemmy to repeatedly process that chain, leading to stack overflows, federation worker exhaustion and timeouts. Simply put, parts of the server were crashing, too many tasks piled up at once, and requests started timing out and failing to load

You may have see this on the website with 502 errors or the lemmy error screen, and on apps it may have presented you with API timeout errors or "Lemmy is starting" errors.

For a visual representation, this graph shows the memory drop each time the server restarts:

The flat bit to the left is good, everything is fine. The choppy bit to the right, not so good, everything is not fine.

Usually its a one-off comment causing this crash, however in this case the user spent a good portion of time bumping the thread, and we had to process each one of those, each causing a crash, restarting the server, and then crashing on the next in the queue, and so on.

I did try removing the offending community from Lemmy.zip to prevent this from happening (It's quite common behavior in that community to bump threads I think), however we still process all the activities from that community - the only certain fix for now is to defederate until a version of lemmy is released that fixes this.

The graph is back to improving now:

Hope that all makes sense!

Demigodrick

Did you miss The Button the first time around?

Now is your chance to have a press! Season 2 starts today.

Sign up now at The Button or join in the discussion at !thebutton@lemmy.zip

Yes, The Button is returning for a second season!

This time around, your press resets after 24 hours and you get to press again (if the game makes it that far!)

The chat will open from 10:00 UTC on Sunday, 2 hours before the game begins.

A new feature this time around, in addition to the reset press, is achievements in chat for things like first presser and fastest presser. There may even be an Easter egg to find...

The sun is shining. The leaves are blossoming. Sheep skip happily across the fields.

It is April, my friends.

Server Updates

Thank you to those who took part in my little button experiment - I didn't expect it to go quite so well, and have decided there will be a season 2 of The Button. You can follow along in the new The Button community - !thebutton@lemmy.zip

Piefed.zip has a new admin! Please say hello (and thank you!) to INeedMana who has kindly joined the team to help out with keeping the place clean and tidy!

Lemmy.zip has seen two updates this month, versions 0.19.16 and 0.19.17. These have been smaller updates, while we wait on the 1.0 release to come!

Please Don’t Be a Lurker!

If you’re new here - WELCOME! I hope you’re enjoying your time here :)

Just one small teeny-tiny request. The greatest gift you can give Lemmy.zip and Piefed.zip isn’t money, praise, or a signed copy of your self-published autobiography.. maybe. Its participation.

Upvote the things you like. Start a discussion, debate, or ponder about whether cereal is technically a soup. Make new communities if you don’t see one that fits your oddly specific niche obsession. (We don’t judge. Well, we try not to judge anyway.)

The fediverse naturally ebbs and flows, tides of people come and go. But if you’ve found yourself oddly attached to this strange little corner of the internet? Wonderful. Help it breathe. Help it grow. Help it be just a tiny bit weirder in the best possible way.

So if you’ve gone to the effort of clicking Sign Up and proving you’re not a robot (unless you are, in which case hello and welcome to our new AI overlords), then please, I beg of you:

Stick around. Add your voice. It really does make this place better.

Donations

Want to support us? We have a range of donation options to suit you!

Liberpay (Card, Paypal)

Liberpay is our newest donation method. You can donate with Paypal or with a card. The link is https://liberapay.com/dotzip

OpenCollective (Card)

Our classic donation option, but only supports card payments. The link is https://opencollective.com/lemmyzip

If you're enjoying Lemmy.zip or Piefed.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the sites and keeping the virtual lights on.

Ko-Fi (Card, Paypal)

We also have our Ko-Fi page if you'd rather use this site, which also supports Paypal and Card payments.

Thank you!

We continue to have some really kind and generous donators and I can't express my thanks enough. You can see all the kind donators in the Thank You thread - you could get your name in there too! (I am a little behind with this sorry but I will update it soon!)

Operation Epic Graphing

Piefed.zip

For Piefed, we only store data for 7 days rather than 30 (because historically we had less storage space on the server) so all graphs are 7 days unless otherwise stated.

CPU:

RAM:

Performance as i speak:

Image storage:

Cloudflare overview:

Requests:

Bandwidth:

Visitors:

Lemmy.zip

CPU:

RAM:

Disk space:

Images:

Cloudflare overview:

Requests:

Bandwidth:

Visitors:

Traffic:

Well that's all for now. Take care all and I'll be back in May for some more graphs and all the updates :)

Thanks all

Demigodrick

Almost have a working setup for a season 2 of The Button.

I am thinking 3 presses each, with an hour minimum between each press? Should give plenty of life to the game.

Also curious as to when the community would like to see it run again. I'm mindful it was originally an April Fools thing so I can run it again in a few hours so that everyone can get a go today. Or I can wait a couple of weeks so people don't feel to burned out on it?

If there is a season 3 though, that will probably be a couple of months away to give people a chance to miss playing it!

Hello everyone - I hope you enjoyed the first season of The Button! I certainly did 😀

Below are some stats based on Season 1, which I hope you find interesting!

The Button — Season 1 Final Stats

Overview

Faction Standings

Purple wins by just 0.2% over Green.

Records

Milestones

Top Instances

Hourly Activity

I am working on updating the platform to support multiple seasons with modifiers :) more info soon!

An experiment has begun.

There’s a button.

That’s it. Just a button… and a timer counting down.

Anyone can press it. Everyone sees what happens. And every press changes things forever.

• One shared button
• No undo, no reset
• A collective test of curiosity, restraint… or chaos

Some of you will hold the line. Some of you won’t last five seconds.

Either way, once it’s over… it’s over.

The Button

So the only question is:

Do you press it?

The Button is open to anyone on any federated platform, be that Lemmy, Piefed, Mastodon... even Stegodon!

Any user from any instance can sign in and press the button, but you only get one chance.

To add another layer to this, users can choose which faction to join, and the "winner" is the faction that holds the button the longest amount of time before it eventually runs out.

Initially a press of the button resets the timer to 15 minutes - this may reduce if the game carries on for a while. At the time of this post, there is about 60 minutes until the button expires. The first button press will be available when there's 15 minutes left, i.e. in 45 minutes, but you can sign up now to be ready!

There is also a permanent record of the first pusher :)