2020-03-16T02:53:02  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 65 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-03-16T03:18:54  *** okurz_ is now known as okurz
2020-03-16T04:13:02  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=35 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-03-16T04:44:10  -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 289 * total waiting locks: 140 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-03-16T04:54:11  -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-03-16T08:43:47  *** ldevulder__ is now known as ldevulder
2020-03-16T08:51:03  *** jadamek2 is now known as jadamek
2020-03-16T11:11:56  -heroes-bot- PROBLEM: Current Load on localhost - CRITICAL - load average: 0.72, 9.99, 11.54 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=localhost&service=Current%20Load
2020-03-16T11:36:56  -heroes-bot- RECOVERY: Current Load on localhost - OK - load average: 0.94, 0.91, 2.90 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=localhost&service=Current%20Load
2020-03-16T12:41:58  <kl_eisbaer> lcp: https://graylog.opensuse.org/search?q=source%3A+matrix&rangetype=relative&relative=300
2020-03-16T12:42:16  <kl_eisbaer> lcp: your matrix server is the top logger on graylog, JFYI
2020-03-16T12:43:03  <kl_eisbaer> ~2500 messages in the last 5 minutes ...
2020-03-16T12:43:38  <kl_eisbaer> what makes me wonder is that the service seems to get stopped and started all the time. Is this by intention ?
2020-03-16T12:50:54  <lcp> ah, excellent
2020-03-16T12:51:36  <lcp> I know why, I got an email from discord saying the bot token was revoked
2020-03-16T12:52:04  <lcp> it should be stopped until I replace the token
2020-03-16T12:52:23  <kl_eisbaer> ok, good to know you found the root cause ;-)
2020-03-16T12:53:26  <kl_eisbaer> matrix python3[17823]: RuntimeError: Password resets emails are configured to be sent from this homeserver due to a partial 'email' block. However, the following required keys are missing: email.smtp_host, email.smtp_port
2020-03-16T12:53:39  <kl_eisbaer> there are some more interesting messages....
2020-03-16T12:53:49  <kl_eisbaer> matrix npm[17782]: Mar-16 12:43:03.304 [DiscordAS] #033[33mwarn#033[39m: [DEPRECATED] The user store is now part of the SQL database.The config option userStorePath no longer has any use.
2020-03-16T12:53:53  <lcp> I thought I fixed that
2020-03-16T12:54:05  <lcp> oh wait, maybe somebody run highstate
2020-03-16T12:54:17  <lcp> that is still broken in git
2020-03-16T12:54:36  <kl_eisbaer> jip, this happens to some of my hosts as well. highstate seems to be running very often and regular now
2020-03-16T12:55:19  <lcp> since gitlab pipeline is broken, this gets really problematic very fast
2020-03-16T12:55:58  <kl_eisbaer> jip. I'm still waiting for the gitlab heroes to fix this. But it looks like this takes more time than initially expected
2020-03-16T12:56:25  <kl_eisbaer> time to setup gitlab-runner hosts, it seems ... :-/
2020-03-16T12:56:45  <lcp> I will stop synapse now
2020-03-16T12:57:19  <lcp> and this will be fixed with one of the mrs I sent out some time ago
2020-03-16T12:57:25  <kl_eisbaer> could you maybe install the latest kernel update, while you are on it anyway?
2020-03-16T12:57:36  <lcp> oh righty-o
2020-03-16T12:58:26  <kl_eisbaer> thanks!
2020-03-16T12:59:29  <kl_eisbaer> https://monitor.opensuse.org/icingaweb2/ with grafana graphs is also on it's way. The only sad message atm is that check_mk-agent is not really useful any longer for icinga2 setups :-/
2020-03-16T13:00:26  <kl_eisbaer> So I'm currently re-thinking all the check scripts on the hosts...
2020-03-16T13:01:18  <kl_eisbaer> using Salt to deploy them is easy. But this way, we loose the automatic inventory feature of check_mk :-(
2020-03-16T13:04:13  <lcp> yup
2020-03-16T13:05:09  <lcp> alright, it's updated and synapse and discord are stopped
2020-03-16T13:05:15  <lcp> at least until the next highstate
2020-03-16T13:07:16  <kl_eisbaer> lcp: let's wait and see .... :-)
2020-03-16T13:07:49  <kl_eisbaer> lcp: JFYI: you should be able to log in to https://graylog.opensuse.org already and use the search to see the logs remotely
2020-03-16T13:09:48  <lcp> yeah, I was just looking, it seems like nothing's going on anymore ;)