admiralpatrick

Mostly posting this to see if it jump starts federation, but the rant applies.

It's like a bi-weekly event where LW just randomly stops sending traffic to random instances for 2-3 days. I give up. If it federates, it federates. Gonna just start divesting from communities there.

Testing the post reply muting feature in Tesseract. I'm setting this post to mute replies, so hopefully any comments will be silently marked as read without triggering a notification on my end.

My kitchen is from the 50s and has been updated somewhat over the years by previous owners. The wiring has all be updated to Romex but it's still all running from two circuits, and one of them is inconveniently placed and practically useless. The end result is I can only use one countertop appliance at a time without tripping a breaker. Only the dishwasher and oven have dedicated circuits.

I've lived with this limitation long enough. My 2026 project is to put each outlet on its own circuit and move a couple other outlets from circuits that are shared with adjacent rooms. In all, it's looking like it's going to be 5 or 6 total circuits.

Would I be ahead to do a single big circuit (220V split phase) from the breaker box and break it out in a sub-panel in the kitchen or just run new individual circuits up from the main breaker box?

Secondary question:

Assuming I do the sub panel and break out five 15 amp circuits in the kitchen, that's 75 amps. I only have 100A service from the meter. I do not ever anticipate drawing 75 amps from the kitchen outlets at once, but AFAIK codes require that I account for the possibility.

Would it meet code (NEC) to put a 30 amp "main" breaker on the sub-panel that feeds 75 amps worth of 15 amp circuits (or, alternatively, feed the sub-panel from a 30 amp breaker in the main panel)?

Wanted to throw out a Halloween song for all the music communities I mod but was coming up blank for the 70s, so this will have to do.

During some work with Tess, I'd notice that my test instance was running horribly slow. The CPU was spiking, Postgres was not happy and using pretty much all the available compute.

Investigating, I found the culprit to be some crawler or possibly malicious actor sending a massive number of unscoped requests to /api/v3/comment/list. What I mean by "unscoped" is without limiting it to a post ID. I'm not sure if this is a bug in Lemmy or there's a legit use for just fetching only comments outside of a post, but I digress as that's another discussion.

After disallowing unscoped requests to the comment list endpoint (see mitigation further down), no more issue.

The kicker seemed to be that this bot / jackass was searching by "Old" and was requesting thousands of pages deep.

Requests looked like this: GET /api/v3/comment/list?limit=50&sort=Old&page=16413

Since I shutdown Dubvee officially, I'm not keeping logs as long as I used to, but I saw other page numbers in the access log, but they were all above 10,000. From the logs I have available, the requests seem to be coming from these 3 IP addresses, but I have insufficient data to confirm this is all of them (probably isn't).

• 134.19.178.167
• 213.152.162.5
• 134.19.179.211

Log Excerpt

Note that I log the query string as well as the URI. I've run a custom Nginx setup for so long, I actually don't recall if the query string is logged by default or not. If you're not logging the query string, you can still look for the 3 (known) IPs above making requests to /api/v3/comment/list and see if entries similar to these show up.

2025-09-21T14:31:59-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:00-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:01-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:01-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:12-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:13-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:13-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"
2025-09-21T14:32:13-04:00 {LB_NAME}: dubvee.org, https, {LB_IP}, 134.19.179.211, - , NL, Amsterdam, North Holland, 52.37590, 4.89750, TLSv1.3, TLS_AES_256_GCM_SHA384, "GET", "/api/v3/comment/list", "limit=50&sort=Old&page=16413"

Mitigation:

First, I blocked the IPs making these requests, but they would come back from a different one. Finally, I implemented a more robust solution.

My final mitigation was to simply reject requests to /api/v3/comment/list that did not have a post ID in the query parameters. I did this by creating a dedicated location block in Nginx that is an exact match for /api/v3/comment/list and doing the checks there.

I could probably add another check to see if the page number is beyond a reasonable number, but since I'm not sure what, if any, clients utilize this, I'm content just blocking unscoped comment list requests entirely. If you have more info / better suggestion, leave it in the comments.

# Basically an and/or for has post_id or has saved_only
map $has_post_id:$has_saved_only $comment_list_invalid{
  "1:0"	1;
  "0:1" 1;
  "1:1" 1;
  default 0;
}

server {

...

location = /api/v3/comment/list {

  # You'll need the standard proxy_pass headers such as Host, etc. I load those from an include file.
  include conf.d/includes/http/server/location/proxy.conf;

  # Create a variable to hold a 0/1 state
  set $has_post_id 0;

  # If the URL query string contains 'post_id' set the variable to 1
  if ($arg_post_id) {
    set $has_post_id  1;
  }
  if ($arg_saved_only) {
    set $has_saved_only 1;
  }

  # If the comment_list_invalid map resolves to 0, "send" a 444 resposne
  # 444 is an Nginx-specific return code that immediately closes the connection 
  # and wastes no further resources on the request
  if ($comment_list_invalid = 0) {
    return 444;
  }

  # Otherwise, proxy pass to the API as normal 
  # (replace this with whatever your upstream name is for the Lemmy API
  proxy_pass "http://lemmy-be/";
}

This release has feature-creeped quite a bit, but the end result is turning out pretty great. I also promise it's not vaporware and really does exist 😆

Hopefully, within the next 1-2 weeks I will have enough of it put back together and polished to throw out a beta release to the hosted instance for general use. Would love to get some extra eyes on the new features and suss out any bugs / UX stumbling blocks.

What's New Since Last Update?

• The sidebar and community menu (in the top navbar) can now browse/search all communities in addition to your subscribed/grouped/favorites. You can even scope the search to local communities only -- right from the sidebar.

• Re-implemented the form fields. Made them look all fancy-pantsy with integrated icons/buttons

• Spruced up the login, forgot password, sign up, and instance selection pages

• The search and community browser now take your filters into account. If you've filtered / blocked an instance, for example, nothing from that instance will show up in the search results nor will communities from there be listed if you're browsing/searching for communities.

• "Trending Posts' widget in the right-hand sidebar. Shows the "Top Hour" posts.

  • Uses your default feed setting for local, subscribed or all
  • Takes your filters into account.
  • Anything you've downvoted will not show up in "Trending"
  • Posts in the trending list are randomized each load rather than sorting by score
  • Updates every 10 minutes

-Can filter/hide removed posts, downvoted posts, posts/comment below a set score

• Mentions and DMs can be disabled. When disabled, they will not be fetched or shown in the inbox, and when it checks for notifications, it will silently mark them as read and not show in the notification count. You can still see them in other clients (i.e. they're only hidden in Tesseract), but they will be marked as read. I think I mentioned this before, but at the time, I did not have the notification suppression working; that's integrated now.

• Added a button bar every ~50 posts in the feed with some helpful actions:

  • Scroll to top of feed
  • Refresh Feed
  • Truncate feed to only posts below the bar

• Why would you want to truncate the feed periodically?

  1. To free up memory, especially if you're deep into infinite scrolling
  2. Performance: When navigating away from the feed, it takes and stores a snapshot of the current posts and loads + re-renders them when you come back. Truncating the feed reduces the amount of posts it has to re-render when it loads.
  3. Performance: Any time you update group memberships or modify the filter policy (or enable/disable filtering), every post in the feed has to react to that change and re-run the filter against itself. When you truncate the feed, the truncated posts are no longer rendered and thus do not have to react to events.
  4. A version of this behavior used to be automatic, but it would randomly glitch out in Firefox in ways I could never replicate consistently in order to properly fix it. Doing it manually, it can avert the issue by scrolling to the top. It's less jarring for it to scroll to the top when it's invoked manually versus automatically.

• Lots more little bug fixes for existing bugs and new ones introduced during this dev cycle

• Probably more I'm not thinking of.

Notable changes:

• Made the secondary toolbar able to be hidden (shown hidden here)
• Moved the instance site card above the feed like it normally is on mobile
  • Moved the taglines into the card itself instead of in a separate card below it.
• Added the site search to the top navbar
• Added a "Trending Posts" widget to the sidebar
  • Shows the results for "Top 6 Hours" based on your user setting for your default feed (Local, All, or Subscribed)
  • Takes your filters into account: if you've filtered certain people, communities, keywords, etc, those will not show in the trending list

Search Enhancements

The search has been revamped and no longer tries to be "too much". I took out the infinite scroll, moved the search scope filters out of the secondary navbar and into the page, and greatly simplifed the logic. And it works so much better. It also has the option to search for post URLs now.

The site/community search bar at the top of the app has also been enhanced.

• Will detect activity pub URLs, resolve them, and take you directly to the content while skipping the actual search. This lets you quickly load a user/comment/community/post from a federated link. (e.g. paste in an actor ID for a user, and it takes you straight to their profile page. Paste in a post activity pub URL, and it will take you to the local copy of that post.)

  • Also works with both Lemmyverse and Threadiverse links for users, communities, posts, and comments. Haven't added LemShare yet since it uses a different format. Also not sure how widely-used LemShare is, so may not add support.

• Search prefixes to set the search type. You can now prefix the search query to set the search scope from the search box:

  • user: {query}: Searches only for users with the given query string
  • users: {query: Alias for user:
  • community: {query}: Searches only for communities with the given query string
  • communities: {query]: Alias for community:
  • post: {query}: Searches only for posts with the given query string
  • posts: {query}: Alias for post:
  • comment: {query}: Searches only for comments with the given query string
  • comments: {query}: Alias for comment:
  • url: {query}: Searches only for posts which have the provided URL as the post link

Example Search Queries:

• https://example.com/post/12345 will automatically resolve and then take you directly to the local copy of that post.

• https://example.com/u/alice Will automatically resolve Alice and then take you directly to her profile page.

• users: bob: Will search for users that have "bob" in their username or display name.

• community: Dogs Searches for any community that has "dogs" in its name or display name.

• posts: New York City Searches only for posts containing New York City

• comments: New York City Searches only for comments containing "New York City"

• url: https://example.com/foo/bar.htm Searches for any posts that link to exactly https://example.com/foo/bar.htm