2025-10-26T01:29:27  <matrix-o-o> <Emma [it/its]> yaeh seems to be rock solid though, i guses the latest release just bumped memory usage just enough to bump into oomkiller
2025-10-26T05:54:53  *** teepee_ is now known as teepee
2025-10-26T14:55:07  *** teepee_ is now known as teepee
2025-10-26T21:46:55  <matrix-o-o> <Emma [it/its]> acidsys do we care about synapse logging to disk at all?
2025-10-26T21:47:18  <matrix-o-o> <Emma [it/its]> i'd expect the log aggregation service to work fine with systemd journaled logging, right?
2025-10-26T22:07:47  <acidsys> Emma: yes, syslog wise it's actually easier as I do not need to configure another file to pick up
2025-10-26T23:02:31  <matrix-o-o> <Emma [it/its]> oh youre saying if i made a salt PR i should just drop file logging outright?
2025-10-26T23:03:56  <acidsys> I was just answering regarding "work fine"
2025-10-26T23:04:18  <acidsys> generally I prefer if it logs to the journal but in this case would be good if the amount of output was reduced too
2025-10-26T23:14:49  <matrix-o-o> <Emma [it/its]> the problem is that the current output is.. nothing
2025-10-26T23:15:50  <acidsys> I'm confused, are we not talking about /var/log/matrix-synapse/homeserver.log ?
2025-10-26T23:18:37  <matrix-o-o> <Emma [it/its]> yes
2025-10-26T23:19:06  <acidsys> then what do you mean the output is "nothing"? there are lines printed to this file faster than I can read
2025-10-26T23:19:33  <matrix-o-o> <Emma [it/its]> oh wow that file actually contains logs 👀
2025-10-26T23:19:56  <acidsys> and so do homeserver.log.{1,2,3}
2025-10-26T23:20:32  <acidsys> funnily one would expect that it's rotating the files but in reality all of them get a constant flow of output
2025-10-26T23:20:59  <matrix-o-o> <Emma [it/its]> but yeah, even homeserver.log is too quiet
2025-10-26T23:21:15  <matrix-o-o> <Emma [it/its]> ... ironically
2025-10-26T23:22:40  <matrix-o-o> <Emma [it/its]> also seeing a lot of database errors 👀
2025-10-26T23:23:36  <matrix-o-o> <Emma [it/its]> ah its failing to process metrics... odd
2025-10-26T23:24:55  <acidsys> postgresql server is constantly printing errors related to matrix too
2025-10-26T23:25:49  <matrix-o-o> <Emma [it/its]> that's quite concerning
2025-10-26T23:27:23  <matrix-o-o> <Emma [it/its]> i wonder if it's synapse being dumb or whether something has gone wrong with postgres
2025-10-26T23:27:31  <acidsys> https://paste.opensuse.org/pastes/8fea9d593bb6
2025-10-26T23:28:06  <matrix-o-o> <Emma [it/its]> "Oct 26 23:27:29 matrix salt-minion[336166]: [CRITICAL] The payload signature did not validate."
2025-10-26T23:28:08  <matrix-o-o> noticed this in journalctl
2025-10-26T23:28:09  <acidsys> I also grep for ERROR that is not matrix and there is nothing (ok, one line about unique constriant in gitlab)
2025-10-26T23:28:40  <acidsys> yeah salt-minion is broken on tumbleweed right now :-( => https://bugzilla.opensuse.org/show_bug.cgi?id=1251776
2025-10-26T23:30:23  <matrix-o-o> <Emma [it/its]> ah rip
2025-10-26T23:30:49  <matrix-o-o> <Emma [it/its]> anyways, getting corrupted postgres index vibes from this