Saidit.net - Jan 14, 2020 - New updates, automatic title fetching, comment history lookback limit bug fixed

submitted by magnora7[A] from self.SaidIt

Hey everyone, you may have noticed some occasional temporary down time over the last 16 hours or so. It was just d3rr and myself, working on fixing bugs.

First bug that was fixed: As per the responses on the thread we made, I improved the title fetching. Now when you paste a URL in to the URL box when making a post, a title will automatically generate (unless you already have a title typed in, in which case you can hit the 'suggest title' button like normal to overwrite it). Next, we're working on improving the functionality of the title fetch itself, but this will come later. I've diagnosed the problem with why the title fetch itself fails so often, now we just have to write a functioning solution.

Lastly and most importantly, d3rr was able to fix a gigantic bug with our database. For a while now there has been an artificial limit on how many comments it would let you look back in comment history, for some reason when viewing a user's or sub's history or comment pages, the comments would stop fetching after 1000 comments. If you had muted subs included, this number returned would be even smaller. This ended up being a setting deep in our caching. Now it is fixed so that the number of comments that can be viewed is unlimited. This is some bottleneck we inherited from the reddit open source that d3rr had to fix. However the fix requires the entire comments cache to be rebuilt from scratch, so that's why everyone's comments may have gone briefly missing a few hours ago. We should be all good now though. Kudos to d3rr! This was a tough bug to crack.

It's exciting to see saidit continue to grow and improve. Thanks to everyone who helped diagnose and report these bugs, we appreciate it!

all 15 comments
sorted by:
top
newinsightfulfun

[–]useless_aether 9 insightful - 2 fun9 insightful - 1 fun10 insightful - 2 fun -  (6 children)

thank you guys for making this site into such a well performing tool. i think the unlimited history alone makes saidit more useful than the competition.

[–]magnora7[S] 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (5 children)

No problem.

i think the unlimited history alone makes saidit more useful than the competition.

I agree. We intend to create something of historical record here, that will be as permanent as the internet allows. And full comment lookback is an important part of that historical record being accessible, so I couldn't be happier that d3rr got it working properly.

[–]useless_aether 4 insightful - 1 fun4 insightful - 0 fun5 insightful - 1 fun -  (4 children)

truly amazing. lucky, that text compresses extremely well! i just hope the search function works too, i almost never use it, but now i think i should.

[–]go1dfish 5 insightful - 1 fun5 insightful - 0 fun6 insightful - 1 fun -  (1 child)

I don’t think the Reddit/saidit codebase is set up for such compression.

I’ve switched notabug.io over to using gzip encryption on the live db in keeping with the aim of cheap operation. All of saidit’s content and nab’s content with indexes fits in 4gb of disk space this way while operating. Before these optimizations it was taking 15Gb on disk.

My indexing system in its current form is not well suited for historical access though, I use a limit of 1000 items for most listings to keep performance reasonable. This can likely be improved in the future with some sort of hierarchical date based indexing.

[–]JasonCarswell 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (0 children)

Date based indexing might also be good for incremental sharding, should you choose to share backups, decentralize, etc. rather than large complete-full-archive torrents.

(I'm talking abstractly out my ass in general concepts and have no idea how to practically do this in code.)

[–]magnora7[S] 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (1 child)

It works pretty well, but it's not perfect. The search has lots of options you can view beneath the search bar, to help you refine your search.

And if all else fails, you can just use google or duckduckgo with the qualifier "site:saidit.net" as part of your search.

[–]useless_aether 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (0 children)

site:saidit.net

ah yes, should def use that. thanks again.

[–]sproketboy 4 insightful - 3 fun4 insightful - 2 fun5 insightful - 3 fun -  (0 children)

Very nice! Thanks!

[–]SierraKiloBravo 4 insightful - 2 fun4 insightful - 1 fun5 insightful - 2 fun -  (0 children)

:-)

[–]JasonCarswell 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (1 child)

A+++

Keep on hitting the important tweaks and the age and origin of the code isn't important. Like a vintage roadster, some of your favourite highschool tunes, a woman who's hot at any age, or the Millennium Falcon - some classics just stay cool with a modicum of upkeep.

[–]macadoum 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (0 children)

Well said.

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (3 children)

Was trying to link to this: grim reaper)? In comments.

The hyperlink contains a parenthesis at the end of the URL but the formatting can't differentiate between ending parenthesis of URL and the formatting for it in the comment. Henceforth the URL doesn't address properly.

Example: [grim reaper](https://en.wikipedia.org/wiki/Death_(personification))

Probably not be a big deal. But just thought I'd give you a heads up on it in case it was important for any future issues.

Thanks!

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Oh I see, interesting. Thanks for pointing that out, I'll add it to our to-do list

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

Oh actually here is how you do it, you need what's called an "escape character" that tells it not to regard that parenthesis as meaningful. Our escape character is \

grim reaper

So this will work, ending the link in Death_(personification\)) The \ tells it to ignore the first ) as a finishing statement to the link, so then it uses the second one as it should.

We might still automate this for this instance, but that's how you fix this issue. Thanks

[–][deleted] 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (0 children)

Great! Now I can finish up my other comment!

Keyboards clacking