2018-12-28T00:00:41 RECOVERY: Hosts syslog on monitor.infra.opensuse.org - OK: Tested /var/log/opensuse/hosts/ - no files older than 240 minutes found ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-12-28T00:02:58 *** Son_Goku has quit IRC 2018-12-28T00:03:27 *** King_InuYasha has joined #opensuse-admin 2018-12-28T00:10:48 *** Son_Goku has joined #opensuse-admin 2018-12-28T00:11:16 *** King_InuYasha has quit IRC 2018-12-28T00:16:25 *** King_InuYasha has joined #opensuse-admin 2018-12-28T00:17:31 *** Son_Goku has quit IRC 2018-12-28T00:24:43 *** Son_Goku has joined #opensuse-admin 2018-12-28T00:25:02 *** King_InuYasha has quit IRC 2018-12-28T00:30:40 *** Son_Goku has quit IRC 2018-12-28T00:35:00 *** Son_Goku has joined #opensuse-admin 2018-12-28T00:48:04 *** King_InuYasha has joined #opensuse-admin 2018-12-28T00:50:28 *** Son_Goku has quit IRC 2018-12-28T00:54:52 *** King_InuYasha has quit IRC 2018-12-28T00:55:38 *** Son_Goku has joined #opensuse-admin 2018-12-28T01:00:53 *** Son_Goku has quit IRC 2018-12-28T01:04:24 *** King_InuYasha has joined #opensuse-admin 2018-12-28T01:10:36 *** Son_Goku has joined #opensuse-admin 2018-12-28T01:11:56 *** King_InuYasha has quit IRC 2018-12-28T01:18:08 *** Son_Goku has quit IRC 2018-12-28T01:20:38 *** Sir_Gallantmon has joined #opensuse-admin 2018-12-28T01:22:35 *** Son_Goku has joined #opensuse-admin 2018-12-28T01:25:58 *** Sir_Gallantmon has quit IRC 2018-12-28T01:29:36 *** King_InuYasha has joined #opensuse-admin 2018-12-28T01:31:27 *** Son_Goku has quit IRC 2018-12-28T01:34:51 *** Son_Goku has joined #opensuse-admin 2018-12-28T01:37:04 *** King_InuYasha has quit IRC 2018-12-28T01:41:14 *** Sir_Gallantmon has joined #opensuse-admin 2018-12-28T01:42:24 *** Son_Goku has quit IRC 2018-12-28T01:46:37 *** Sir_Gallantmon has quit IRC 2018-12-28T01:52:36 *** Sir_Gallantmon has joined #opensuse-admin 2018-12-28T01:57:07 *** Sir_Gallantmon has quit IRC 2018-12-28T02:26:38 *** lcp has quit IRC 2018-12-28T02:38:34 *** Son_Goku has joined #opensuse-admin 2018-12-28T02:43:40 *** King_InuYasha has joined #opensuse-admin 2018-12-28T02:46:24 *** Son_Goku has quit IRC 2018-12-28T02:50:21 *** Son_Goku has joined #opensuse-admin 2018-12-28T02:50:53 *** King_InuYasha has quit IRC 2018-12-28T02:57:27 *** Son_Goku has quit IRC 2018-12-28T02:58:35 *** King_InuYasha has joined #opensuse-admin 2018-12-28T03:03:26 *** Son_Goku has joined #opensuse-admin 2018-12-28T03:04:33 *** King_InuYasha has quit IRC 2018-12-28T03:08:43 *** King_InuYasha has joined #opensuse-admin 2018-12-28T03:10:30 *** Son_Goku has quit IRC 2018-12-28T03:12:24 *** Son_Goku has joined #opensuse-admin 2018-12-28T03:15:10 *** King_InuYasha has quit IRC 2018-12-28T03:19:21 *** King_InuYasha has joined #opensuse-admin 2018-12-28T03:20:03 *** Son_Goku has quit IRC 2018-12-28T03:21:33 *** Son_Goku has joined #opensuse-admin 2018-12-28T03:24:07 *** King_InuYasha has quit IRC 2018-12-28T03:26:02 *** King_InuYasha has joined #opensuse-admin 2018-12-28T03:28:52 *** Son_Goku has quit IRC 2018-12-28T03:31:53 *** Son_Goku has joined #opensuse-admin 2018-12-28T03:31:54 *** okurz has quit IRC 2018-12-28T03:33:46 *** King_InuYasha has quit IRC 2018-12-28T03:34:03 *** okurz has joined #opensuse-admin 2018-12-28T03:38:50 *** Son_Goku has quit IRC 2018-12-28T03:38:56 *** King_InuYasha has joined #opensuse-admin 2018-12-28T03:42:03 *** boombatower has joined #opensuse-admin 2018-12-28T03:42:12 *** Son_Goku has joined #opensuse-admin 2018-12-28T03:45:10 *** King_InuYasha has quit IRC 2018-12-28T03:52:00 *** King_InuYasha has joined #opensuse-admin 2018-12-28T03:52:42 *** Son_Goku has quit IRC 2018-12-28T03:57:39 *** Son_Goku has joined #opensuse-admin 2018-12-28T03:57:58 *** King_InuYasha has quit IRC 2018-12-28T04:10:58 *** King_InuYasha has joined #opensuse-admin 2018-12-28T04:13:05 *** Son_Goku has quit IRC 2018-12-28T04:40:09 *** boombatower has quit IRC 2018-12-28T06:56:23 PROBLEM: Root Partition on localhost - DISK CRITICAL - free space: / 2099 MB (10% inode=71%): ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=localhost&service=Root%20Partition 2018-12-28T07:27:18 PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 47465752 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby 2018-12-28T07:47:18 RECOVERY: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY OK: DB mb_opensuse2 (host:mirrordb2) 131432 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby 2018-12-28T09:58:05 *** tigerfoot has quit IRC 2018-12-28T10:00:42 *** tigerfoot has joined #opensuse-admin 2018-12-28T10:20:19 PROBLEM: PerfData on monitor.infra.opensuse.org - CRITICAL: Found files older than 2880 minutes /var/lib/pnp4nagios/perfdata//proc_Rsyslog_Process.xml was last modified on Thu Jun 21 09:42:54 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=PerfData 2018-12-28T10:32:03 *** kl_eisbaer has joined #opensuse-admin 2018-12-28T10:35:11 JFYI: some machines are down because one of the hypervisors is down at the moment. 2018-12-28T10:35:16 I count 2018-12-28T10:35:19 icc.opensuse.org 2018-12-28T10:35:23 narwal2.opensuse.org 2018-12-28T10:35:41 mirrordb1 (one DB server behind download.opensuse.org) 2018-12-28T10:35:56 gcc-stats.opensuse.org and one or two additional infra machines 2018-12-28T10:36:39 I will try to get the VMs restarted, but it would be cool if someone from the current SUSE Admins would take over, as I'm not familiar any longer with the setup 2018-12-28T10:52:38 PROBLEM: mongodb queries per second on icc.infra.opensuse.org - CRITICAL - General MongoDB Error: could not find master/primary ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=icc.infra.opensuse.org&service=mongodb%20queries%20per%20second 2018-12-28T10:56:35 PROBLEM: mongodb connections on icc.infra.opensuse.org - CRITICAL - General MongoDB Error: could not find master/primary ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=icc.infra.opensuse.org&service=mongodb%20connections 2018-12-28T10:59:36 OK: duke, icc, mirrordb1 and gcc-stats are up again now 2018-12-28T11:00:36 narwal2 has a boot problem (Control-D to continue) - and I don't know the root-pw. So I can not help here, sorry 2018-12-28T11:02:02 more or less the same applies for duke and icc - the machines are up, but I don't know if everything is back to normal on them. So leaving this up to the guys who should know.... 2018-12-28T11:08:21 *** cboltz has joined #opensuse-admin 2018-12-28T12:26:12 *** cboltz has quit IRC 2018-12-28T13:06:32 *** ldevulder has joined #opensuse-admin 2018-12-28T13:26:04 *** lcp has joined #opensuse-admin 2018-12-28T14:53:42 PROBLEM: SSH on nuka.infra.opensuse.org - SSH CRITICAL - OpenSSH_7.6 (protocol 2.0) version mismatch, expected OpenSSH_7.2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nuka.infra.opensuse.org&service=SSH 2018-12-28T15:42:07 RECOVERY: SSH on minnie.infra.opensuse.org - SSH OK - OpenSSH_7.6 (protocol 2.0) ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=minnie.infra.opensuse.org&service=SSH 2018-12-28T15:44:30 RECOVERY: NRPE on minnie.infra.opensuse.org - NRPE v3.2.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=minnie.infra.opensuse.org&service=NRPE 2018-12-28T15:48:31 *** King_InuYasha is now known as Son_Goku 2018-12-28T16:03:47 *** ldevulder has quit IRC 2018-12-28T16:19:11 *** cboltz has joined #opensuse-admin 2018-12-28T16:19:36 *** ldevulder has joined #opensuse-admin 2018-12-28T16:23:45 *** tigerfoot has quit IRC 2018-12-28T16:24:21 PROBLEM: keepalived on elsa.infra.opensuse.org - Became master: 0 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=keepalived 2018-12-28T16:25:27 *** King_InuYasha has joined #opensuse-admin 2018-12-28T16:26:16 *** Son_Goku has quit IRC 2018-12-28T16:39:27 *** King_InuYasha has quit IRC 2018-12-28T16:40:01 *** Son_Goku has joined #opensuse-admin 2018-12-28T16:45:17 PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 2980952 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby 2018-12-28T16:49:00 *** King_InuYasha has joined #opensuse-admin 2018-12-28T16:50:07 *** Son_Goku has quit IRC 2018-12-28T16:53:37 *** Son_Goku has joined #opensuse-admin 2018-12-28T16:54:52 *** King_InuYasha has quit IRC 2018-12-28T16:57:35 *** King_InuYasha has joined #opensuse-admin 2018-12-28T16:59:59 *** Son_Goku has quit IRC 2018-12-28T17:09:34 *** Son_Goku has joined #opensuse-admin 2018-12-28T17:11:22 *** King_InuYasha has quit IRC 2018-12-28T17:39:05 *** heroes-bot has joined #opensuse-admin 2018-12-28T17:40:39 *** heroes-bot has joined #opensuse-admin 2018-12-28T17:45:18 *** heroes-bot has joined #opensuse-admin