2020-05-31T12:55:27  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 52 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-05-31T13:13:09  <bmwiedemann1> is download.o.o down?
2020-05-31T13:13:35  <cboltz> I think so :-(
2020-05-31T13:14:17  <cboltz> I can't even ping it (tested public and internal IPs)
2020-05-31T13:17:28  <bmwiedemann1> looks pretty dead on VNC and serial, too
2020-05-31T13:18:07  <cboltz> any last words^Wmessages?
2020-05-31T13:18:27  <cboltz> if yes, save them somewhere ;-)
2020-05-31T13:18:34  <bmwiedemann1> nothing I could see. Did force-reset, booting now.
2020-05-31T13:18:54  <cboltz> ok, thanks
2020-05-31T13:19:10  <cboltz> ssh works again :-)
2020-05-31T13:19:55  <bmwiedemann1> We sometimes do libvirt live-migration when we have to reboot hosts and I have that feeling as if something breaks whenever we live-migrate
2020-05-31T13:20:02  <cboltz> and the webserver is also back
2020-05-31T13:23:44  <bmwiedemann1> journal does not have anything from before reboot, but /var/log/messages had last entry 2020-05-31T12:52:00.431266+00:00  and then jumps straight to 2020-05-31T13:18:29 without anything in between
2020-05-31T13:24:17  <cboltz> right, I also just checked
2020-05-31T13:25:28  <bmwiedemann1> cboltz: audit.log has various DENIED messages
2020-05-31T13:25:41  <bmwiedemann1> from apparmor
2020-05-31T13:26:22  <cboltz> looks like the check_iostat monitoring plugin needs some additional permissions ;-)
2020-05-31T13:27:48  <cboltz> ah, it "just" needs permissions for reading /etc/passwd and /etc/nsswitch.conf
2020-05-31T13:33:46  <cboltz> created request id 810470
2020-05-31T19:11:09  <bmwiedemann1> for the record in case hardware problems are involved, pontifex2 VM ran on atreju3 host today when it crashed
2020-05-31T22:36:03  *** DrDro[m]10 is now known as DrDro[m]12
2020-05-31T22:36:04  *** DrDro[m]12 is now known as DrDro[m]14