2020-03-29T00:02:25  -heroes-bot- RECOVERY: NRPE on nue-ns2.infra.opensuse.org - NRPE v3.2.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nue-ns2.infra.opensuse.org&service=NRPE
2020-03-29T00:02:28  -heroes-bot- RECOVERY: NRPE on svn.infra.opensuse.org - NRPE v3.2.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=svn.infra.opensuse.org&service=NRPE
2020-03-29T00:02:54  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=47 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-03-29T00:13:04  -heroes-bot- PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 27.794667 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-03-29T00:22:37  -heroes-bot- PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service galera2 is DOWN on galera proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy
2020-03-29T00:22:39  -heroes-bot- PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service galera2 is DOWN on galera proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy
2020-03-29T00:23:59  -heroes-bot- PROBLEM: MySQL WSREP recv on galera3.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 9.529723 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera3.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-03-29T00:24:53  -heroes-bot- PROBLEM: MySQL WSREP recv on galera2.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 109.052434 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera2.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-03-29T00:27:26  -heroes-bot- RECOVERY: HAProxy on anna.infra.opensuse.org - HAPROXY OK -  elections-test (Active: 1/1) pinot (Active: 1/1) monitor_graylog (Active: 1/1) kubic (Active: 1/1) mirrorlist (Active: 1/1) metrics (Active: 1/1) osccollab (Active: 1/1) smt (Active: 1/1) mickey (Active: 1/1) redmine_test (Active: 1/1) download (Active: 1/1) community2 (Active: 1/1) lists (Active: 1/1) riesling (Active: 1/1) monitor (Active: 1/1) elections (Active: 1/1) rsync-community2 (Active: 0/1) studioexpress (Active: 3/3) status (Active: 1/1) jekyll (Active: 1/1) galera (Active: 1/1 , Backup: 2/2) etherpad (Active: 1/1) svn (Active: 1/1) community (Active: 1/1) lnt (Active: 1/1) forums (Active: 1/1) monitor_grafana (Active: 1/1) freeipa (Active: 1/1) jenkins (Active: 1/1) static (Active: 3/3) connect (Active: 1/1) obsreview (Active: 1/1) github_101 (Active: 0/1) rpmlint (Active: 1/1) redmine (Active: 1/1) html5test (Active: 3/3) gccstats (Active: 1/1) nuka (Active: 1/1) download-private (Active: 1/1) kernel-git-in (Active: 0/1) dale (Active: 1/1) shop (Active: 3/3) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy
2020-03-29T00:27:28  -heroes-bot- RECOVERY: HAProxy on elsa.infra.opensuse.org - HAPROXY OK -  monitor_graylog (Active: 1/1) dale (Active: 1/1) shop (Active: 3/3) monitor_grafana (Active: 1/1) forums (Active: 1/1) elections-test (Active: 1/1) gccstats (Active: 1/1) elections (Active: 1/1) lnt (Active: 1/1) riesling (Active: 1/1) download (Active: 1/1) mirrorlist (Active: 1/1) galera (Active: 1/1 , Backup: 2/2) mickey (Active: 1/1) status (Active: 1/1) jenkins (Active: 1/1) pinot (Active: 1/1) community2 (Active: 1/1) html5test (Active: 3/3) redmine_test (Active: 1/1) svn (Active: 1/1) connect (Active: 1/1) freeipa (Active: 1/1) monitor (Active: 1/1) jekyll (Active: 1/1) smt (Active: 1/1) rsync-community2 (Active: 0/1) static (Active: 3/3) community (Active: 1/1) nuka (Active: 1/1) metrics (Active: 1/1) github_101 (Active: 0/1) rpmlint (Active: 1/1) kernel-git-in (Active: 0/1) kubic (Active: 1/1) etherpad (Active: 1/1) obsreview (Active: 1/1) osccollab (Active: 1/1) lists (Active: 1/1) redmine (Active: 1/1) download-private (Active: 1/1) studioexpress (Active: 3/3) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy
2020-03-29T03:01:51  *** okurz_ is now known as okurz
2020-03-29T06:04:09  -heroes-bot- RECOVERY: MySQL WSREP recv on galera1.infra.opensuse.org - OK wsrep_local_recv_queue_avg = 0.489396 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-03-29T08:14:00  -heroes-bot- RECOVERY: MySQL WSREP recv on galera3.infra.opensuse.org - OK wsrep_local_recv_queue_avg = 0.487516 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera3.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-03-29T15:32:54  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 79 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-03-29T17:22:54  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=37 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-03-29T20:03:05  -heroes-bot- PROBLEM: HAProxy on provo-proxy1.infra.opensuse.org - HAPROXY CRITICAL - Active service debuginfod is DOWN on debuginfod proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=provo-proxy1.infra.opensuse.org&service=HAProxy