2018-08-17T00:01:41 *** Son_Goku has joined #opensuse-admin 2018-08-17T00:34:35 *** tigerfoot has quit IRC 2018-08-17T00:36:27 *** nicolasbock has quit IRC 2018-08-17T00:58:49 *** Son_Goku has quit IRC 2018-08-17T01:46:06 *** Son_Goku has joined #opensuse-admin 2018-08-17T01:59:22 *** Fraser_Bell has quit IRC 2018-08-17T02:09:56 *** Son_Goku has quit IRC 2018-08-17T03:56:01 *** lcp has quit IRC 2018-08-17T04:20:42 cboltz: I've been out and about all day, but I'm the morning I will :) 2018-08-17T05:25:37 tampakrap: coming back to what foursixnine wrote yesterday around lunch time we still have bigger problems with o3 as documented in https://progress.opensuse.org/issues/39743 . Some requests – during some times many if not all – respond with 504 but checking /var/log/apache2/access_log on ariel we only find a few 502, 503, 504 so we assume that the HA proxy is answering more requests with 504 than it should. Can we get a 2018-08-17T05:25:38 hold of any logs from the HA proxy? What is the timeout setting? Can we see *which* requests are answered in this way by the proxy? 2018-08-17T05:29:21 *** malcolmlewis has quit IRC 2018-08-17T06:56:07 *** |miska| has quit IRC 2018-08-17T06:56:41 *** |miska| has joined #opensuse-admin 2018-08-17T07:29:41 *** tigerfoot has joined #opensuse-admin 2018-08-17T08:14:18 *** Son_Goku has joined #opensuse-admin 2018-08-17T08:54:36 *** enavarro has joined #opensuse-admin 2018-08-17T09:35:41 *** thomic has joined #opensuse-admin 2018-08-17T09:35:46 *** thomic has left #opensuse-admin 2018-08-17T09:36:29 *** thomic has joined #opensuse-admin 2018-08-17T09:41:05 *** cboltz has joined #opensuse-admin 2018-08-17T10:05:50 PROBLEM: PerfData on monitor.infra.opensuse.org - CRITICAL: Found files older than 2880 minutes /var/lib/pnp4nagios/perfdata//proc_Rsyslog_Process.xml was last modified on Thu Jun 21 09:42:54 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=PerfData 2018-08-17T10:05:59 *** lcp has joined #opensuse-admin 2018-08-17T10:11:56 *** lcp has quit IRC 2018-08-17T10:34:33 *** Son_Goku has quit IRC 2018-08-17T10:35:47 *** Son_Goku has joined #opensuse-admin 2018-08-17T10:35:57 *** nicolasbock has joined #opensuse-admin 2018-08-17T10:56:56 *** Son_Goku has quit IRC 2018-08-17T10:57:34 *** Son_Goku has joined #opensuse-admin 2018-08-17T11:00:39 *** Son_Goku has quit IRC 2018-08-17T11:57:55 *** Son_Goku has joined #opensuse-admin 2018-08-17T12:01:27 foursixnine / okurz: https://progress.opensuse.org/issues/39743#note-20 2018-08-17T12:02:11 tampakrap: thanks :) 2018-08-17T12:12:24 *** malcolmlewis has joined #opensuse-admin 2018-08-17T12:38:43 *** a-865k has quit IRC 2018-08-17T12:51:01 who is minnie.i.o.o and why is the salt master on o3 trying to reach it? 2018-08-17T12:52:10 minnie is the salt master in the heroes network 2018-08-17T12:52:37 (no idea about the second half of your question) 2018-08-17T12:54:01 I guess someone took over the salt master config from other machines in i.o.o to o3 but not change it after a change to SUSE-DMZ something… 2018-08-17T12:54:10 Will create ticket as it's not urgent 2018-08-17T12:54:30 okurz: ariel can't access minnie 2018-08-17T12:54:58 yes, I get it. and it shouldn't, right? Should I just manually adjust o3:/etc/salt/minion.d/f_defaults.confthen? 2018-08-17T12:55:06 I meant /etc/salt/minion.d/f_defaults.conf :) 2018-08-17T12:55:13 you can just remove the salt minion 2018-08-17T12:55:19 there is no salt master on the suse-dmz 2018-08-17T12:55:51 tampakrap: o3 *is* a salt master 2018-08-17T12:57:02 ah cool 2018-08-17T12:57:32 The file says it's managed by salt but I doubt that this is true anymore, more likely *was* managed by salt by another master but isn't anymore, right? 2018-08-17T12:57:38 it is not true any more 2018-08-17T12:57:50 So delete the line from the header and replace by "master: localhost"? I will check if it works for sure 2018-08-17T12:58:12 yes 2018-08-17T12:58:48 tampakrap: And I should clarify that salt on o3 is merely used for interactive remote execution only, not actual application of recipes ;) So `salt '*' cmd.run 'foo'` only :) 2018-08-17T12:59:41 okay 2018-08-17T13:00:03 you will probably have to adjust the certificates as well, it has already accepted the certificate of the old saltmaster 2018-08-17T13:00:13 so you'll probably have to delete it and let it accept the new one 2018-08-17T13:03:10 done 2018-08-17T13:03:45 so one thing left is to change the domain from infra.o.o to suse-dmz.o.o but let's not do it on a friday afternoon 2018-08-17T13:05:13 Yes, I have noted your comment but you are right :) 2018-08-17T13:09:17 * cboltz already knows a list of "things that break if you change the domain" from infra.o.o, and would pay to hear similar stories about renaming suse-dmz.o.o ;-) 2018-08-17T13:11:59 there are a bunch of services already renamed to suse-dmz.o.o already 2018-08-17T13:20:09 *** Son_Goku has quit IRC 2018-08-17T13:28:22 thomic: so … restart of o3 for more cores, I would like to schedule that *not* now as we have too much eyes and attention on o3 now. We could do it over the weekend or start of next week maybe, WDYT? 2018-08-17T13:31:03 *** Son_Goku has joined #opensuse-admin 2018-08-17T13:40:49 okurz: whatever you like 2018-08-17T13:54:53 I will not ask you to do over the weekend which people might not consider nice or fair ;) only asking to not do it on a Friday afternoon. Let's try to not forget about it and select a time when e.g. the next Tumbleweed snapshot is released. Shortly after that should be always a better time :) 2018-08-17T14:12:35 good that I always tend to copy-paste textfield entries before submitting them. Trying to update the incident entry on status.opensuse.org for o3 gave me a 500 internal error :D 2018-08-17T14:13:13 anyone wants to check the remote side on status.opensuse.org for error code cc9f5a7f-58b8-4b88-b8fe-6f1017d10173 ? 2018-08-17T14:15:39 reproducible and can not seem to be able to update it with any content. 2018-08-17T14:16:52 how to reproduce? 2018-08-17T14:18:30 tampakrap: can you try to update https://status.opensuse.org/dashboard/incidents/56/updates ? 2018-08-17T14:19:02 to fixed? 2018-08-17T14:19:31 no, try "watching" 2018-08-17T14:19:46 needs message 2018-08-17T14:20:09 ok, write "system stabilized, monitoring. More details in https://progress.opensuse.org/issues/39743" 2018-08-17T14:20:58 reproduced! 2018-08-17T14:26:02 okurz: nothing at the apache logs 2018-08-17T14:29:16 okurz: I don't want to touch the apache configs now to increase logging level, remind me on monday please 2018-08-17T14:29:28 meanwhile you can edit the incident instead of posting an update https://status.opensuse.org/dashboard/incidents/56 2018-08-17T14:33:42 *** Son_Goku has quit IRC 2018-08-17T14:54:15 *** Son_Goku has joined #opensuse-admin 2018-08-17T14:56:51 *** Son_Goku has quit IRC 2018-08-17T15:16:55 *** Son_Goku has joined #opensuse-admin 2018-08-17T15:46:53 *** Son_Goku has quit IRC 2018-08-17T15:51:04 *** thomic has quit IRC 2018-08-17T15:51:54 *** Son_Goku has joined #opensuse-admin 2018-08-17T15:55:58 *** thomic has joined #opensuse-admin 2018-08-17T16:19:09 *** Son_Goku has quit IRC 2018-08-17T16:27:59 *** enavarro has quit IRC 2018-08-17T16:51:31 *** lcp has joined #opensuse-admin 2018-08-17T17:10:34 okurz, tampakrap this is a known issue - but nevertheless, I'd like to ask you to (also) use the broken update feature ;-) 2018-08-17T17:10:36 reason: 2018-08-17T17:10:54 "update" sends out a mail to subscribers before going 500 2018-08-17T17:11:19 afterwards, edit the initial issue and add your updated text 2018-08-17T17:21:23 *** a-865k has joined #opensuse-admin 2018-08-17T17:29:05 *** a-865k has quit IRC 2018-08-17T17:32:10 *** Son_Goku has joined #opensuse-admin 2018-08-17T17:36:52 *** a-865k has joined #opensuse-admin 2018-08-17T17:38:06 *** a-865k has quit IRC 2018-08-17T17:39:22 cboltz: so then I must have sent like 10 announcements ;) 2018-08-17T17:46:28 *** Son_Goku has quit IRC 2018-08-17T17:58:04 *** a-865k has joined #opensuse-admin 2018-08-17T18:11:11 okurz: better than none ;-) 2018-08-17T18:42:16 *** MrMazda has joined #opensuse-admin 2018-08-17T18:44:36 *** a-865k has quit IRC 2018-08-17T18:47:30 *** a-865k has joined #opensuse-admin 2018-08-17T18:55:57 *** MrMazda has quit IRC 2018-08-17T18:58:20 *** a-865k has quit IRC 2018-08-17T19:04:22 *** MrMazda has joined #opensuse-admin 2018-08-17T19:14:05 *** MrMazda has quit IRC 2018-08-17T19:42:24 *** Son_Goku has joined #opensuse-admin 2018-08-17T19:45:17 *** MrMazda has joined #opensuse-admin 2018-08-17T19:50:46 *** nicolasbock has quit IRC 2018-08-17T19:52:28 *** Fraser_Bell has joined #opensuse-admin 2018-08-17T19:54:36 *** Son_Goku has joined #opensuse-admin 2018-08-17T19:55:02 *** nicolasbock has joined #opensuse-admin 2018-08-17T20:27:12 *** Son_Goku has quit IRC 2018-08-17T20:33:16 *** Son_Goku has joined #opensuse-admin 2018-08-17T20:45:08 *** Fraser_Bell has quit IRC 2018-08-17T21:02:59 *** Son_Goku has quit IRC 2018-08-17T21:18:06 *** |miska| has quit IRC 2018-08-17T21:20:50 PROBLEM: Hosts syslog on monitor.infra.opensuse.org - CRITICAL: Found files older than 600 minutes /var/log/opensuse/hosts//simba.log was last modified on Fri Aug 17 11:15:01 2018 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-08-17T21:59:40 *** nicolasbock has quit IRC 2018-08-17T22:00:50 RECOVERY: Hosts syslog on monitor.infra.opensuse.org - OK: Tested /var/log/opensuse/hosts/ - no files older than 240 minutes found ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=monitor.infra.opensuse.org&service=Hosts%20syslog 2018-08-17T22:27:56 *** Son_Goku has joined #opensuse-admin 2018-08-17T22:50:05 *** cboltz has quit IRC 2018-08-17T23:14:03 *** a-865k has joined #opensuse-admin