2018-12-20T00:06:00  *** Fraser_Bell has joined #opensuse-admin
2018-12-20T00:06:41  <heroes-bot> RECOVERY: Hosts syslog on monitor.infra.opensuse.org - OK: Tested /var/log/opensuse/hosts/ - no files older than 240 minutes found ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog
2018-12-20T00:52:45  *** Fraser_Bell has quit IRC
2018-12-20T03:26:58  *** okurz has quit IRC
2018-12-20T03:30:00  *** okurz has joined #opensuse-admin
2018-12-20T05:07:01  *** boombatower has quit IRC
2018-12-20T05:18:09  *** ldevulder_ has joined #opensuse-admin
2018-12-20T05:21:07  *** ldevulder has quit IRC
2018-12-20T06:37:06  *** lcp has joined #opensuse-admin
2018-12-20T07:27:18  <heroes-bot> PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 3017408 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2018-12-20T07:27:34  *** ldevulder_ is now known as ldevulder
2018-12-20T08:21:24  <heroes-bot> PROBLEM: Root Partition on localhost - DISK CRITICAL - free space: / 1928 MB (10% inode=71%): ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=localhost&service=Root%20Partition
2018-12-20T08:30:53  *** lcp has quit IRC
2018-12-20T09:01:02  *** mcaj has joined #opensuse-admin
2018-12-20T09:32:44  <heroes-bot> PROBLEM: SSL cert status2 on status2.opensuse.org - CRITICAL - Certificate status.opensuse.org expired on Sat 13 Oct 2018 09:10:13 AM GMT +0000. ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status2.opensuse.org&service=SSL%20cert%20status2
2018-12-20T09:45:03  *** ldevulder has quit IRC
2018-12-20T09:53:01  *** ldevulder has joined #opensuse-admin
2018-12-20T09:57:02  *** ldevulder has quit IRC
2018-12-20T10:20:19  <heroes-bot> PROBLEM: PerfData on monitor.infra.opensuse.org - CRITICAL: Found files older than 2880 minutes /var/lib/pnp4nagios/perfdata//proc_Rsyslog_Process.xml was last modified on Thu Jun 21 09:42:54 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=PerfData
2018-12-20T10:42:58  *** matthias_bgg has joined #opensuse-admin
2018-12-20T11:02:18  *** ldevulder has joined #opensuse-admin
2018-12-20T11:10:03  *** lcp has joined #opensuse-admin
2018-12-20T12:28:41  *** cboltz has joined #opensuse-admin
2018-12-20T12:41:11  *** lcp has quit IRC
2018-12-20T12:41:35  *** lcp has joined #opensuse-admin
2018-12-20T12:58:21  *** lcp_ has joined #opensuse-admin
2018-12-20T13:01:43  *** lcp has quit IRC
2018-12-20T13:01:43  *** lcp_ is now known as lcp
2018-12-20T14:27:11  <cboltz> FYI: I'm just installing kernel updates on riesling and water, expect a short wiki downtime in the next minutes
2018-12-20T14:31:06  <cboltz> done
2018-12-20T14:31:42  <cboltz> now over to status2 and provo-mirror...
2018-12-20T14:36:48  <cboltz> done
2018-12-20T14:37:34  <cboltz> tampakrap: the SSL certificate on status2 is expired - do you know how it should get updated?
2018-12-20T14:39:12  <cboltz> (also updating and rebooting status.o.o now)
2018-12-20T14:40:09  <cboltz> (done)
2018-12-20T14:46:27  *** matthias_bgg has quit IRC
2018-12-20T14:53:43  <heroes-bot> PROBLEM: SSH on nuka.infra.opensuse.org - SSH CRITICAL - OpenSSH_7.6 (protocol 2.0) version mismatch, expected OpenSSH_7.2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nuka.infra.opensuse.org&service=SSH
2018-12-20T14:56:01  <tampakrap> copy it from anna or from status1
2018-12-20T14:56:05  <tampakrap> it's wildcard
2018-12-20T15:00:32  <cboltz> the cert on status1 is _not_ wildcard, it's only valid for status.o.o and status1.o.o
2018-12-20T15:01:09  <tampakrap> replace that one then as well :)
2018-12-20T15:01:17  <cboltz> I'd guess status and status2 run dehydrated on their own
2018-12-20T15:01:23  <cboltz> at least the cert file owner indicates that ;-)
2018-12-20T15:01:43  <tampakrap> it's not needed any more
2018-12-20T15:01:51  <tampakrap> since we have the wildcard now
2018-12-20T15:02:20  <cboltz> agreed in general - but I doubt "copy around the certificate" is the solution we want ;-)
2018-12-20T15:02:44  <tampakrap> so you can update the cert and update the hooks.sh on crtmgr
2018-12-20T15:02:45  <heroes-bot> RECOVERY: SSL cert status2 on status2.opensuse.org - OK - Certificate status.opensuse.org will expire on Tue 05 Feb 2019 02:25:17 AM GMT +0000. ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status2.opensuse.org&service=SSL%20cert%20status2
2018-12-20T15:05:00  <cboltz> no, I can't - crtmgr doesn't let me log in (thomic did this intentionally when he set up that machine - IIRC because it's also used for other networks or something like that)
2018-12-20T15:05:20  <cboltz> so you or thomic will have to do that certificate stuff
2018-12-20T15:06:42  <tampakrap> file me a ticket then
2018-12-20T15:06:53  <cboltz> ok
2018-12-20T15:07:08  <cboltz> for now, I reverted the cert on status2.o.o - expired is still better than wrong domain ;-)
2018-12-20T15:07:34  <tampakrap> what wrong domain?
2018-12-20T15:07:45  <tampakrap> take the wildcard from anna and put it on both status[1-2]
2018-12-20T15:15:35  <tampakrap> and stop the dehydrated on statsu[1-2] please
2018-12-20T15:15:44  <tampakrap> when does the wildcard cert expire?
2018-12-20T15:16:45  <heroes-bot> PROBLEM: SSL cert status2 on status2.opensuse.org - CRITICAL - Certificate status.opensuse.org expired on Sat 13 Oct 2018 09:10:13 AM GMT +0000. ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status2.opensuse.org&service=SSL%20cert%20status2
2018-12-20T15:26:45  <cboltz> Feb 5
2018-12-20T15:29:53  *** lcp has quit IRC
2018-12-20T15:30:21  *** lcp has joined #opensuse-admin
2018-12-20T15:33:02  *** lcp_ has joined #opensuse-admin
2018-12-20T15:33:03  <cboltz> I copied the cert from anna to status2 - works for now
2018-12-20T15:34:52  <cboltz> dehydrated.timer was _not_ enabled, and I also didn't find a cronjob, so I'll assume for now that the renewal wasn't automated
2018-12-20T15:35:20  <cboltz> i will _not_ touch the cert on status1 because automatic renewal works there without problems
2018-12-20T15:35:43  *** lcp_ has quit IRC
2018-12-20T15:35:46  <cboltz> you can change that when you adjust crtmgr to deliver the wildcard cert to status{1,2}
2018-12-20T15:36:42  *** lcp has quit IRC
2018-12-20T15:36:45  <heroes-bot> RECOVERY: SSL cert status2 on status2.opensuse.org - OK - Certificate opensuse.org will expire on Mon 04 Feb 2019 11:00:42 PM GMT +0000. ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=status2.opensuse.org&service=SSL%20cert%20status2
2018-12-20T15:37:05  *** lcp_ has joined #opensuse-admin
2018-12-20T15:37:59  *** lcp_ is now known as lcp
2018-12-20T15:40:51  *** lcp has quit IRC
2018-12-20T15:48:04  *** antlarr has quit IRC
2018-12-20T16:01:03  <tampakrap> okay!
2018-12-20T16:01:53  *** antlarr has joined #opensuse-admin
2018-12-20T16:02:06  <cboltz> BTW: just in case something breaks - will you be around during christmas, or at least be reachable via signal?
2018-12-20T16:14:57  *** dddh_ has quit IRC
2018-12-20T16:15:36  *** dddh_ has joined #opensuse-admin
2018-12-20T16:15:51  <heroes-bot> PROBLEM: keepalived on elsa.infra.opensuse.org - Became master: 0 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=keepalived
2018-12-20T16:22:56  *** cboltz has quit IRC
2018-12-20T16:42:40  *** lcp has joined #opensuse-admin
2018-12-20T16:55:58  *** matthias_bgg has joined #opensuse-admin
2018-12-20T17:11:35  <thomic> aehm tampakrap cboltz if anything needed regarding crtmgr ... tell me
2018-12-20T17:12:01  <thomic> for now it's deploying a lot of certs, that's why i did not expose it :)
2018-12-20T17:12:19  <thomic> and because it does DNS updates on the MF dns net
2018-12-20T17:12:36  <thomic> and i think there were some extra hosts
2018-12-20T17:12:51  <thomic> however, we could work on getting this in a better state
2018-12-20T18:28:19  *** matthias_bgg has quit IRC
2018-12-20T19:11:00  *** lcp_ has joined #opensuse-admin
2018-12-20T19:11:24  *** lcp has quit IRC
2018-12-20T19:12:12  *** lcp_ is now known as lcp
2018-12-20T21:34:41  <heroes-bot> PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//simba.log was last modified on Thu Dec 20 11:26:06 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog
2018-12-20T21:43:12  *** cboltz has joined #opensuse-admin
2018-12-20T21:52:39  *** lcp has quit IRC
2018-12-20T21:53:05  *** lcp has joined #opensuse-admin
2018-12-20T22:07:18  <heroes-bot> PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 3356272 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2018-12-20T22:15:25  *** lcp has quit IRC
2018-12-20T22:17:44  *** lcp_ has joined #opensuse-admin
2018-12-20T22:20:34  *** lcp_ is now known as lcp
2018-12-20T22:27:17  <heroes-bot> RECOVERY: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY OK: DB mb_opensuse2 (host:mirrordb2) 562280 and 0 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby
2018-12-20T22:30:34  *** lcp has quit IRC
2018-12-20T22:31:15  *** lcp has joined #opensuse-admin
2018-12-20T22:55:02  *** lcp has quit IRC
2018-12-20T22:55:18  *** lcp_ has joined #opensuse-admin
2018-12-20T22:56:03  *** cboltz has quit IRC
2018-12-20T23:02:47  *** Son_Goku has quit IRC
2018-12-20T23:06:08  *** boombatower has joined #opensuse-admin
2018-12-20T23:16:26  *** antlarr has quit IRC
2018-12-20T23:30:34  *** lcp_ has quit IRC
2018-12-20T23:31:00  *** antlarr has joined #opensuse-admin