2026-06-21T07:23:46  <knurpht> chat.o.o is down. FYI.
2026-06-21T11:05:45  <acidsys> right, matrix.i.o.o seems dead
2026-06-21T11:06:07  <acidsys> reset now
2026-06-21T11:07:56  <rorysys> i wonder what happened, i wasnt able to reach that server for a bit
2026-06-21T11:08:22  <acidsys> machine was locked up, after reset it came back
2026-06-21T11:08:31  <rorysys> im a bit late but got distraced with damage control somewhere else
2026-06-21T11:08:39  <rorysys> ah i see... blegh
2026-06-21T11:09:03  <rorysys> hm, can you try updating the synapse service to use a cache_factor of 3 or 4 rather than 2 to see if that helps smooth out the OOM issue?
2026-06-21T11:09:37  <rorysys> given that from discussion the running theory is that its OOMing under spurious loads where cache misses are causing extra allocations
2026-06-21T11:10:36  <acidsys> according to documentation default is 0.5 and we don't have it set
2026-06-21T11:11:31  <acidsys> added caches: global_factor: 4
2026-06-21T11:22:42  <rorysys> i did find a definition in the systemd service stating 2, fwiw
2026-06-21T11:23:19  <rorysys> unsure if synapse even honors that anymore given ive never heard of it before
2026-06-21T12:24:52  <acidsys> oh I see. removed
2026-06-21T13:41:24  <rorysys> alright
2026-06-21T13:59:46  <acidsys> also solved the other outstanding alerts (for now..): again broken setup on stonehat, certificates for slc-monitor, reboot of squanchy, updates for discourse01/mailman, mail queue on status1, code-dev mysql errors, sample-app service
2026-06-21T14:20:16  <acidsys> and matrix memory alert is back :)
2026-06-21T14:29:46  <matrix-o-o> <Emma [it/its]> i wonder what this looks like from monitoring.o.o
2026-06-21T14:29:59  <matrix-o-o> <Emma [it/its]> oh huh, i guess the irc bridge doesnt even post links to media or whatever
2026-06-21T14:30:08  <matrix-o-o> <Emma [it/its]> there's 2 python processes sitting at 7gb
2026-06-21T17:58:32  <acidsys> Emma: do we know why it's consuming more memory lately? if it's legitimate, we can also increase resources ...
2026-06-21T18:04:54  <matrix-o-o> <Emma [it/its]> latest running theory is another homeserver implementaiton bug
2026-06-21T18:05:26  <matrix-o-o> <Emma [it/its]> where it just spams requests and doesnt acknowledge most of them, was fixed last week or so but instances still might not have updated yet
2026-06-21T18:06:14  <acidsys> oh I see
2026-06-21T19:47:13  <acidsys> Notes for 16.0 -> 15.6 downgrade: 1. the old procps ships symlinks in /sbin, but if /sbin was already converted to a symlink to /usr/sbin, sysctl will not be installed correctly; needs manual extraction of the binary. 2. rpm queries will print a warning about the database format, which is not a problem for zypper, but trips pkg.* functions in salt (made
2026-06-21T19:47:15  <acidsys> https://github.com/saltstack/salt/pull/69503)
2026-06-21T21:21:18  <matrix-o-o> <alarrosa> Hi, my wife is trying to register to openSUSE conference and when she signs in into events.opensuse.org firefox always shows an error "The page isn't redirecting properly" and if I open the developer tools in the browser, I can see over 20 requests getting a 302 response from the server to https://events.opensuse.org/ . I tried with my user and I have no problem signing in, but with hers, it always enters into the loop...
2026-06-21T21:21:24  <matrix-o-o> ... of 302 status codes (I tried on different computers and cleaning the cookies and server data stored in the browser). Also, she can log in to id.opensuse.org fine. Any idea of what to try?
2026-06-21T21:22:53  <matrix-o-o> <alarrosa> btw, she created her user a few days ago but didn't verify her email right after registering. Maybe that left her user in a bad state?
2026-06-21T21:39:26  <matrix-o-o> <Emma [it/its]> interesting, that doesnt seem to happen to me
2026-06-21T21:41:41  <acidsys> hi, this is a known problem with this software :-( https://github.com/openSUSE/osem/issues/3587
2026-06-21T21:44:06  <acidsys> if you share the username, I can check. send with a ticket if you prefer to not post it here.