2018-01-07T00:01:34 *** plusky has quit IRC 2018-01-07T00:07:53 *** cboltz has quit IRC 2018-01-07T00:11:04 *** |miska| has quit IRC 2018-01-07T00:12:53 *** cthugha has quit IRC 2018-01-07T00:15:55 *** plusky has joined #opensuse-admin 2018-01-07T00:17:17 *** paka has quit IRC 2018-01-07T00:17:33 *** paka has joined #opensuse-admin 2018-01-07T00:17:35 *** fvogt has quit IRC 2018-01-07T00:29:50 *** [0x0a] has joined #opensuse-admin 2018-01-07T00:54:22 *** Fraser_Bell has joined #opensuse-admin 2018-01-07T00:54:22 *** Fraser_Bell has joined #opensuse-admin 2018-01-07T01:04:36 *** Son_Goku has joined #opensuse-admin 2018-01-07T01:41:36 *** |miska| has joined #opensuse-admin 2018-01-07T01:41:37 *** |miska| has joined #opensuse-admin 2018-01-07T01:52:16 *** Fraser_Bell has quit IRC 2018-01-07T02:08:27 *** [0x0a] has quit IRC 2018-01-07T02:13:36 *** |miska| has quit IRC 2018-01-07T02:14:35 *** |miska| has joined #opensuse-admin 2018-01-07T02:14:35 *** |miska| has joined #opensuse-admin 2018-01-07T02:43:56 *** Son_Goku has quit IRC 2018-01-07T03:32:41 *** Son_Goku has joined #opensuse-admin 2018-01-07T03:37:39 *** okurz has quit IRC 2018-01-07T03:38:06 *** okurz_ has joined #opensuse-admin 2018-01-07T03:38:13 *** okurz_ is now known as okurz 2018-01-07T04:28:12 PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//wikisearch.log was last modified on Wed Jan 3 18:24:51 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-01-07T05:16:22 *** paka has quit IRC 2018-01-07T05:16:22 *** paka has joined #opensuse-admin 2018-01-07T09:12:06 *** tigerfoot has joined #opensuse-admin 2018-01-07T09:30:26 *** Son_Goku has quit IRC 2018-01-07T09:39:16 *** a-865-kx has quit IRC 2018-01-07T10:02:26 *** a-865 has joined #opensuse-admin 2018-01-07T11:02:24 *** cboltz has joined #opensuse-admin 2018-01-07T11:10:25 *** fvogt has joined #opensuse-admin 2018-01-07T11:42:21 *** cboltz_ has joined #opensuse-admin 2018-01-07T11:42:21 *** cboltz_ has joined #opensuse-admin 2018-01-07T11:45:03 *** cboltz has quit IRC 2018-01-07T11:49:54 *** cboltz_ is now known as cboltz 2018-01-07T12:14:48 tampakrap: handling status.o.o via haproxy is a bad idea 2018-01-07T12:15:09 I just removed status2.o.o from the haproxy config so that only status1 gets used 2018-01-07T12:15:42 background: status*.o.o intentionally uses a local mysql, and the database doesn't get synced between status1 and status2 2018-01-07T12:15:55 so if haproxy randomly redirects to one of them, we get a brain split 2018-01-07T12:16:32 it is not via haproxy 2018-01-07T12:16:51 status2.opensuse.org has address 130.57.72.11 2018-01-07T12:17:16 yes, but status.opensuse.org is an alias for proxy.opensuse.org. 2018-01-07T12:17:34 ah 2018-01-07T12:18:00 I noticed the problem as a follow-up of the galera downtime 2018-01-07T12:18:26 I wanted to update status.o.o to mark the services as running again, and surprisingly they were already green 2018-01-07T12:18:44 and the message about the broken galera cluster didn't exist 2018-01-07T12:18:53 okay I'll change the DNS 2018-01-07T12:19:10 it turned out haproxy had redirected me to status2 when I marked the services as down 2018-01-07T12:21:50 any objections to installing the kernel update on riesling and water *now*? 2018-01-07T12:22:07 feel free 2018-01-07T12:22:31 ok, starting on water as usual 2018-01-07T12:22:44 status1 doesn't have a public IP 2018-01-07T12:23:06 ah no it has, sorry 2018-01-07T12:23:13 actually it has only public IP 2018-01-07T12:23:33 and that's what the haproxy is using as well 2018-01-07T12:24:35 (the reason for asking before I do kernel updates is that thomic told me a while ago that reboots hang in rare cases, and that I'll need someone who can force-reboot the VM in that case) 2018-01-07T12:26:42 status.o.o is now a cname to status1.o.o 2018-01-07T12:26:56 better, thanks! 2018-01-07T12:27:17 riesling and water updated and rebooted, wiki works as expected 2018-01-07T12:28:48 RECOVERY: memcached data stored on riesling.infra.opensuse.org - MEMCACHE OK: memcached 1.4.33 on 127.0.0.1:11211, up 2 minutes 42 seconds - bytes is 2493363 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=riesling.infra.opensuse.org&service=memcached%20data%20stored 2018-01-07T12:35:43 any news about the "random" downtimes in Provo we noticed some weeks ago? 2018-01-07T12:35:52 provo-mirror is down again... 2018-01-07T12:36:57 bryce1 went awol and one colleague from provo looked at it, upgraded the bios and said it is better now 2018-01-07T12:37:11 then bryce2 had the same issue, and I think my colleague updated the bios on that machine again 2018-01-07T12:37:24 s/again/as well 2018-01-07T12:38:11 which one is provo-mirror? 2018-01-07T12:38:13 pontifex? 2018-01-07T12:38:57 pontifex mufasa are both up 2018-01-07T12:39:08 no idea on which machine it's [not] running 2018-01-07T12:39:29 in case it helps, the public IP is 130.57.72.10 2018-01-07T12:39:31 yep it is up 2018-01-07T12:45:14 I can only tell you that http://provo-mirror.opensuse.org is down 2018-01-07T12:46:52 I'll restart it 2018-01-07T13:00:25 *** afaerber has left #opensuse-admin 2018-01-07T13:06:48 still down... 2018-01-07T13:08:36 it works here 2018-01-07T13:10:03 now it works 2018-01-07T13:14:36 monitoring says provo-mirror mailq has 1614 mails 2018-01-07T13:15:16 looks like most of them are from mirror@localhost and to Lars 2018-01-07T13:15:46 I don't understand why I can't ssh to proxy-prv.o.o 2018-01-07T13:15:51 MF-IT closed the ports again? 2018-01-07T13:16:11 ssh -A -p2222 status2 2018-01-07T13:16:24 and from there to provo-mirror -p2222 2018-01-07T13:16:45 the interesting thing about the mailq is that those mails are from Jan 4, 5, 6, 7 2018-01-07T13:17:02 and mailq says 2018-01-07T13:17:05 (delivery temporarily suspended: Host or domain name not found. Name service error for name=suse.de type=MX: Host not found, try again) 2018-01-07T13:18:17 so maybe the machine was up, but the network down/broken? 2018-01-07T13:19:22 yeah try to process it 2018-01-07T13:19:31 hey guys 2018-01-07T13:20:59 I did a postsuper -h ALL to avoid spamming Lars with all those mails, but he probably already got ~200 mails before I could do that 2018-01-07T13:21:32 tampakrap, when you have time, i regretably have to ask you to move my mail alias again because this gmail stuff is driving me crazy, i have to get it off there when you get a chance, please 2018-01-07T13:27:25 katnip: sure, to what? 2018-01-07T13:27:46 james@insiberia.net 2018-01-07T13:30:03 okay done 2018-01-07T13:30:47 thank you very much 2018-01-07T13:31:48 tampakrap: what should we do with the ~1200 mails I set on hold on provo-mirror? 2018-01-07T13:32:40 legally, the only valid option is not to hold them back, but I'm not sure if Lars likes to receive 1200 mails just because that's the legally valid option ;-) 2018-01-07T13:33:26 *** Son_Goku has joined #opensuse-admin 2018-01-07T13:39:00 yeah I agree 2018-01-07T13:55:55 *** fvogt has quit IRC 2018-01-07T14:11:56 so your answer is...? ;-) 2018-01-07T14:13:28 hmm, actually there's another option - consent 2018-01-07T14:13:57 if Lars tells us that to delete those mails, that solves the legal issue 2018-01-07T14:25:43 *** tigerfoot has quit IRC 2018-01-07T14:30:38 *** tigerfoot has joined #opensuse-admin 2018-01-07T15:02:12 *** Son_Goku has quit IRC 2018-01-07T15:06:51 * cboltz wrote a mail to Lars 2018-01-07T16:53:39 *** dddh has quit IRC 2018-01-07T16:57:08 *** Son_Goku has joined #opensuse-admin 2018-01-07T19:02:49 *** Son_Goku has quit IRC 2018-01-07T19:34:32 *** Son_Goku has joined #opensuse-admin 2018-01-07T20:00:03 PROBLEM: HAProxy on mufasa.infra.opensuse.org - HAPROXY CRITICAL - Active service riesling is DOWN on riesling proxy ! Active service narwal4 is DOWN on static proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mufasa.infra.opensuse.org&service=HAProxy 2018-01-07T20:27:48 *** fvogt has joined #opensuse-admin 2018-01-07T20:56:21 *** fvogt has quit IRC 2018-01-07T20:56:45 *** fvogt has joined #opensuse-admin 2018-01-07T21:43:55 *** Fraser_Bell has joined #opensuse-admin 2018-01-07T21:43:55 *** Fraser_Bell has joined #opensuse-admin 2018-01-07T22:06:31 *** malcolmlewis has quit IRC 2018-01-07T22:11:15 *** malcolmlewis has joined #opensuse-admin 2018-01-07T22:11:15 *** malcolmlewis has joined #opensuse-admin 2018-01-07T22:29:56 *** fvogt has quit IRC 2018-01-07T22:44:54 *** Fraser_Bell has quit IRC 2018-01-07T23:32:45 *** cboltz has quit IRC 2018-01-07T23:41:31 *** plinnell has quit IRC 2018-01-07T23:42:49 *** [0x0a] has joined #opensuse-admin 2018-01-07T23:47:32 *** plinnell has joined #opensuse-admin 2018-01-07T23:47:32 *** plinnell has joined #opensuse-admin