Sorry chat is down

submitted by magnora7 from self.SaidIt

We were having a good chat, and it went down. We have to wait for d3rr to fix it because he knows how to restart this part of the software correctly.

all 20 comments
sorted by:
top
newinsightfulfun

[–]Mnemonic 3 insightful - 2 fun3 insightful - 1 fun4 insightful - 2 fun -  (1 child)

Awh man, now I HAVE to go to sleep...

https://www.youtube.com/watch?v=CsRWpW7-skQ

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

Sounds good, I should do the same. Great clip, I love Alan Watts but I haven't heard this one.

[–]Alberto 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (1 child)

All good dude I highly enjoyed chatting with ya'll

[–]magnora7[S] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

same to you man

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (4 children)

/u/d3rr, /u/Mnemonic98, /u/Alberto

[–]Mnemonic 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (3 children)

Not sure if that worked, I discovered it by the new tab. Should we get a message?

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (2 children)

oh yeah I guess you're /u/Mnemonic with no 98

[–]Mnemonic 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

Yeah so this is weird, because I did recieve a reply message, but still don't know if I have to have had a separete thingie for you mentioning me here (which I didn't).

But for some hours now I will not reply anyway.

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

I did another test that isn't a reply to you so you can see how it works. Also the chat is now back up, I think the chat backlog got overloaded with too many link previews or something, we're going to fix it so this doesn't happen again. Thanks for helping give the chat a thorough test so we can work out the bugs haha

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (1 child)

/u/magnora1, /u/Mnemonic test username mention

[–]Mnemonic 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (0 children)

works

[–]magnora7[S] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (7 children)

/u/d3rr I tried restarting everything I could. The instructions at https://saidit.net/s/saiditdev/wiki/admin/chat-admin don't have a way to restart TheLounge without restarting the whole server, which is what we really need. I did a run of

sudo forever start -c "npm start" ./

but that didn't help, and now we might have two instances running. I also ran the user cache clean file we made and it worked for like 2 mins but then chat broke again. I also tried emptying the user folder (except for 1) and that didn't help. Might just need to restart the whole server? It's acting kind of goofy.

After that we need to codify a process to restart TheLounge without restarting everything else. And then perhaps trace why this bug happened... We had been typing a lot and sharing a lot of links. I noticed the link previews stopped loading in the chat before it crashed.

edit: I did nothing for about 10 mins and TheLounge spontaneously started working again. I have no idea what happened. But it couldn't connect. Then I stopped and started unrealircd and everything started working again.

I deleted everything in .lounge/users. I think we should maybe just keep this folder thinned out even more. It seems to be the cause of the instability perhaps? It also seems to strongly affect page load time. Unrealircd also crashed this time though. Not sure if thelounge crashed unrealircd or the other way around... Seems like thelounge was the main culprit though

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (6 children)

Way to keep the ship afloat!

How do we know unreal crashed? I'm hoping it was all TheLounge, maybe trying to do media previews. For sure removing TheLounge config files kicks everyone out of their connected channels, which I think slows page load time and also puts a lot more load on the server. Most of our users probably end up connected to 10 to 20 channels. A periodic full purge here might make sense.

There's kind of a lounge restart procedure, but it is janky and involves killing unix processes. We could experiment with not using forever which would simplify this procedure.

I will make sure we're not running double chat and add documentation for my ghetto TheLounge restart. I also need to update the SSL cert tonight.

Comment expandos are going well, switched to pure js only like Voat and RES, not gonna touch snudown since that stuff goes out for API context where we don't want frontend/browser only extra markup.

[–]magnora7[S] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (5 children)

How do we know unreal crashed?

Well, I don't for absolutely sure know. But I couldn't connect to the server at all once TheLounge was back up. /connect did nothing, and it was acting like the server was unresponsive (but TheLounge was 100% working by this point). So I restarted unrealircd and then it started working fine, so I just assumed unrealircd crashed.

Yeah, I'm wondering if the forever command is what may be causing us problems too. I'm not sure it's meant to be run that way, but I don't know.

Thanks for being on top of the SSL cert, appreciate it.

Pure js sounds like a good approach.

Alright I'm off to do the podcast with Sam Tripoli, wish me luck.

[–][deleted] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (4 children)

Good luck!

Yeah it sounds like unreal did crash.

[–]magnora7[S] 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (3 children)

Boo it got last-minute rescheduled to next week. Oh well :P

But yeah it was weird to see unreal crash. But I only think unreal crashed because TheLounge crashed, because it was definitely the thing to go. The whole lounge box just locked up for everyone at one point and then wouldn't even load on refresh. We were posting lots of links and the image previews had stopped coming up, I noticed. So we hit some cache limit on either the text backlog, the preview image cache, or the user limit we're used to hitting (which would've been quite the coincidence to happen right as we're heavily using it). So I blame the backlogs, either the text or the images or both. I think if we just keep these wiped clean so that users can only go back, say, 100 lines of chat or something, then that'd solve the problem. We probably just need another script that wipes the user folder clean at a certain backlog limit, and maybe on that cleans out the image cache too. Or maybe one that just restarts TheLounge. I would really like to have a script to automate restarting TheLounge and put it in reddit/r2, that'd be hella convenient. I was kind of panicking trying to figure out how to get it restarted, and it was kind of pure luck that it fixed itself (after 10 mins of me giving up and not doing anything, after trying everything), so I'd like to be prepared for that next time. A restart script for TheLounge would streamline this a lot. We could also put an unrealircd restart script in that same folder, and you could combine them to have a "./Restart_chat" script. It's really the only part of the site I don't know how to manually reset without restarting the whole server.

[–]billiam 2 insightful - 1 fun2 insightful - 0 fun3 insightful - 1 fun -  (2 children)

Can I ask why you're even using UnrealIRCd? InspIRCd is so much better.

[–][deleted] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

Hey thanks I'll look into that. It was hard to decide on which server to use not having much IRC experience. Unreal seemed popular and well supported.

[–]magnora7[S] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

We've had zero problems with it, and I think it's more popular. Thanks for the recommendation though, we'll look at it if we have problems.

[–]magnora7[S] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

Chat is back up!