2018-07-24T01:17:03 When I try to access paste.opensuse.org I get a blank white page with just the message, in upper left corner: "Disallowed Key Characters." 2018-07-24T01:34:59 *** Fraser_Bell has quit IRC 2018-07-24T02:46:27 *** okurz has quit IRC 2018-07-24T02:47:07 *** okurz has joined #opensuse-admin 2018-07-24T03:09:19 *** lcp has joined #opensuse-admin 2018-07-24T03:15:57 *** dddh has quit IRC 2018-07-24T03:16:21 *** dddh has joined #opensuse-admin 2018-07-24T03:16:21 *** dddh has joined #opensuse-admin 2018-07-24T04:37:37 *** Son_Goku has joined #opensuse-admin 2018-07-24T05:12:46 *** dddh has quit IRC 2018-07-24T05:13:09 *** dddh has joined #opensuse-admin 2018-07-24T05:13:09 *** dddh has joined #opensuse-admin 2018-07-24T05:19:09 *** plinnell has quit IRC 2018-07-24T05:36:26 *** Son_Goku has quit IRC 2018-07-24T06:20:26 *** plinnell has joined #opensuse-admin 2018-07-24T06:20:27 *** plinnell has joined #opensuse-admin 2018-07-24T06:57:56 *** sven15 has joined #opensuse-admin 2018-07-24T07:06:58 *** duncanmv has joined #opensuse-admin 2018-07-24T07:11:23 *** tigerfoot has quit IRC 2018-07-24T07:13:44 *** tigerfoot has joined #opensuse-admin 2018-07-24T08:06:06 *** ldevulder has joined #opensuse-admin 2018-07-24T08:08:41 *** matthias_bgg has joined #opensuse-admin 2018-07-24T09:56:35 PROBLEM: PerfData on monitor.infra.opensuse.org - CRITICAL: Found files older than 2880 minutes /var/lib/pnp4nagios/perfdata//proc_Rsyslog_Process.xml was last modified on Thu Jun 21 09:42:54 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=PerfData 2018-07-24T10:09:36 *** sven15 has quit IRC 2018-07-24T10:13:48 *** sven15 has joined #opensuse-admin 2018-07-24T10:16:49 *** enavarro has joined #opensuse-admin 2018-07-24T10:21:47 *** cboltz has joined #opensuse-admin 2018-07-24T10:21:47 *** cboltz has joined #opensuse-admin 2018-07-24T10:29:28 *** tigerfoot_ has joined #opensuse-admin 2018-07-24T10:29:50 *** tigerfoot has quit IRC 2018-07-24T10:38:48 *** Son_Goku has joined #opensuse-admin 2018-07-24T11:02:49 *** Son_Goku has quit IRC 2018-07-24T11:21:30 PROBLEM: HTTP rsync.o.o on widehat.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-07-24T11:23:40 *** enavarro has quit IRC 2018-07-24T11:41:21 RECOVERY: HTTP rsync.o.o on widehat.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 1217 bytes in 0.522 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-07-24T11:43:03 *** tigerfoot_ is now known as tigerfoot 2018-07-24T11:44:36 *** Son_Goku has joined #opensuse-admin 2018-07-24T11:46:56 *** duncanmv has quit IRC 2018-07-24T11:50:11 *** duncanmv has joined #opensuse-admin 2018-07-24T11:50:22 *** enavarro has joined #opensuse-admin 2018-07-24T12:17:31 PROBLEM: HTTP rsync.o.o on widehat.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-07-24T13:21:31 *** Son_Goku has quit IRC 2018-07-24T13:21:58 *** Son_Goku has joined #opensuse-admin 2018-07-24T13:26:15 *** duncanmv has quit IRC 2018-07-24T13:53:59 tampakrap: progress is off by 20 minutes vs. world. I am logged in and can "fix" /etc/ntp.conf . Is ntp1.opensuse.org the right one? Also had the same question already a month ago in https://progress.opensuse.org/issues/26832 2018-07-24T13:54:45 I would fix it with salt recipes of course when we have that in salt 2018-07-24T13:54:46 *** Son_Goku has quit IRC 2018-07-24T13:56:39 hm, salt recipes reference ntp1.infra.opensuse.org but not sure if this works 2018-07-24T14:00:04 ok, does not need to be fixed urgently. created https://progress.opensuse.org/issues/38789 and assigned it to myself but I need help to not mess. Which one is the right one… 2018-07-24T14:27:34 *** a-865k has quit IRC 2018-07-24T14:28:50 *** enavarro has quit IRC 2018-07-24T14:42:50 *** a-865k has joined #opensuse-admin 2018-07-24T14:59:16 *** a-865k has quit IRC 2018-07-24T15:08:18 *** a-865 has joined #opensuse-admin 2018-07-24T15:09:53 *** a-865 is now known as a-865k 2018-07-24T15:23:58 *** sven15 has quit IRC 2018-07-24T15:53:15 *** Son_Goku has joined #opensuse-admin 2018-07-24T16:06:57 *** aboe[m] has left #opensuse-admin 2018-07-24T16:14:09 *** ldevulder has quit IRC 2018-07-24T16:21:18 okurz: we have ntp managed with salt, see pillar/salt_cluster/opensuse.sls for the server names 2018-07-24T16:21:39 that said - progress.o.o is one of the old SLE 11 machines, so I won't guarantee for anything ;-) 2018-07-24T16:21:44 *** matthias_bgg has quit IRC 2018-07-24T16:22:07 I'd recommend to run highstate test=True to see what would be changed before really applying it 2018-07-24T16:24:51 cboltz: the salt recipes references ntp1.infra.opensuse.org which did not work for me, ntp1.opensuse.org does 2018-07-24T16:28:26 *** Son_Goku has quit IRC 2018-07-24T16:29:04 indeed, I can confirm (checked on water.infra.o.o) that ntp1.infra.o.o doesn't work :-( 2018-07-24T16:29:16 what about ntp2.infra.o.o and ntp3.infra.o.o? 2018-07-24T16:30:55 *** ldevulder has joined #opensuse-admin 2018-07-24T16:37:32 okurz: I like the title of your merge request ;-)) 2018-07-24T16:38:09 but I don't believe ntp3.o.o works for you - we have ntp{1,2,3}.infra.o.o, but only ntp{1,2}.o.o 2018-07-24T16:49:09 cboltz: fine, I only tested ntp1.o.o ;) 2018-07-24T16:49:57 my understanding is that ntp1.o.o is ntp2.infra.o.o and ntp2.o.o is ntp3.infra.o.o 2018-07-24T16:50:10 ntp1.infra.o.o doesn't have a public counterpart AFAIK 2018-07-24T16:50:11 *** sven15 has joined #opensuse-admin 2018-07-24T16:55:55 yes, but all of this does not explain what I *should* use 2018-07-24T16:56:37 I'd say use ntp2.infra.o.o and/or ntp3.infra.o.o 2018-07-24T16:57:13 then it should have worked in before with progress.o.o using the configuration from the source files but the time has been 20 minutes(!) off 2018-07-24T17:00:38 indeed, that's the theory ;-) 2018-07-24T17:02:21 in practise, we'll probably have to wait for an answer from tampakrap - IIRC he did the ntp stuff and should know what's going on 2018-07-24T17:07:53 hmm, actually I might have found the problem (!= solution) 2018-07-24T17:08:51 *** lcp has quit IRC 2018-07-24T17:09:03 git log c35d260aecbc23ad9fca4f9deeb02ddb2ee56091 (2017-11-08) says that the real ntp servers moved to the SUSE-DMZ network, and that anna.infra.o.o and elsa.infra.o.o were made NTP servers for *.infra.o.o 2018-07-24T17:09:43 and on 2017-12-05 / da9a25096b2719f9825df4ad8152cd125b5bcd93 we switched to chrony on all machines 2018-07-24T17:11:36 I wouldn't be surprised if this broke the server setup... 2018-07-24T17:43:31 PROBLEM: HTTP rsync.o.o on widehat.opensuse.org - CRITICAL - Socket timeout after 10 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-07-24T17:53:27 RECOVERY: HTTP rsync.o.o on widehat.opensuse.org - HTTP OK: HTTP/1.1 200 OK - 1217 bytes in 6.225 second response time ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=widehat.opensuse.org&service=HTTP%20rsync.o.o 2018-07-24T18:08:26 cboltz: "server" as in broke for the old SLE11 progress.o.o or ntp1.infra.o.o as "server"? 2018-07-24T18:08:52 ntp*.infra.o.o as ntp server 2018-07-24T18:09:25 I tried ntpdate ntp{1,2,3}.infra.o.o from a 42.3 VM, and it failed for all of them :-( 2018-07-24T19:23:39 *** Fraser_Bell has joined #opensuse-admin 2018-07-24T19:23:40 *** Fraser_Bell has joined #opensuse-admin 2018-07-24T20:20:27 *** Fraser_Bell has quit IRC 2018-07-24T20:40:02 PROBLEM: HAProxy on elsa.infra.opensuse.org - HAPROXY CRITICAL - Active service dale is DOWN on dale proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=elsa.infra.opensuse.org&service=HAProxy 2018-07-24T20:40:03 PROBLEM: HAProxy on anna.infra.opensuse.org - HAPROXY CRITICAL - Active service dale is DOWN on dale proxy ! ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=anna.infra.opensuse.org&service=HAProxy 2018-07-24T21:36:50 PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//simba.log was last modified on Tue Jul 24 11:29:15 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-07-24T22:06:49 RECOVERY: Hosts syslog on monitor.infra.opensuse.org - OK: Tested /var/log/opensuse/hosts/ - no files older than 240 minutes found ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-07-24T22:16:49 *** cboltz has quit IRC 2018-07-24T22:40:58 *** ldevulder has quit IRC