2019-12-21T00:50:06  <a-865k> now everything works except forums :-(
2019-12-21T01:00:19  <a-865k> 9 minutes later, all 4 forum tabs opened
2019-12-21T02:58:09  *** Eighth_Doctor <Eighth_Doctor!conankudom@fedora/ngompa> has joined #opensuse-admin
2019-12-21T02:58:10  <Eighth_Doctor> I can't access lists.opensuse.org :(
2019-12-21T02:58:20  <Eighth_Doctor> it's showing as unreachable
2019-12-21T02:59:07  *** harrow` <harrow`!~harrow@static.38.6.217.95.clients.your-server.de> has quit IRC (Quit: Leaving)
2019-12-21T03:00:31  *** tux93 <tux93!~irssi@opensuse/member/tux93> has quit IRC (Ping timeout: 250 seconds)
2019-12-21T03:07:35  *** tux93 <tux93!~irssi@opensuse/member/tux93> has joined #opensuse-admin
2019-12-21T03:12:49  *** harrow <harrow!~harrow@static.38.6.217.95.clients.your-server.de> has joined #opensuse-admin
2019-12-21T03:22:00  *** okurz_ <okurz_!~okurz@ppp-46-244-209-106.dynamic.mnet-online.de> has joined #opensuse-admin
2019-12-21T03:24:08  *** okurz <okurz!~okurz@ppp-46-244-200-165.dynamic.mnet-online.de> has quit IRC (Ping timeout: 260 seconds)
2019-12-21T03:24:09  *** okurz_ is now known as okurz
2019-12-21T03:25:15  <a-865k> the long waits are back, minutes reather than seconds
2019-12-21T03:27:03  <a-865k> the weekend and getting screwed by provo is back
2019-12-21T03:28:54  <a-865k> https://en.opensuse.org/Lifetime just loaded sans CSS after multiple minute wait
2019-12-21T03:29:35  <a-865k> now it's waiting on static.opensuse.org
2019-12-21T03:31:36  <a-865k> Network Timeout
2019-12-21T03:31:36  <a-865k> The operation timed out when attempting to contact mirrors.opensuse.org.
2019-12-21T03:36:05  <a-865k> now it's waiting on beans.opensuse.org
2019-12-21T09:00:39  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-12-21T09:16:12  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Ping timeout: 260 seconds)
2019-12-21T09:24:46  *** adrianS <adrianS!~adrian@opensuse/member/AdrianSuSE> has joined #opensuse-admin
2019-12-21T10:09:48  <okurz> a-865k: regarding your network problems: I still have a MF-IT ticket open linked to one of the issues and there is sporadic communication but not really going forward in either way. I agree with kl_eisbaer aka. lrupp who had a good point in https://progress.opensuse.org/issues/52262#note-5 : We are merely acting as proxies and can not really help when there are either individual or sporadic network problems. The services *
2019-12-21T10:09:50  <okurz> are* fine as most of the users at most of the time can interact just fine. I suggest to directly get in contact with someone at microfocus. There are also nice people there but going over multiple hops in communication tends to slow down transfer of information, loose information, loose the "personal touch" and not really reach a conclusion.
2019-12-21T10:18:22  <a-865k> okurz:  Until reading 52262 now I had no idea it was possible to contact anyone at MF "directly".
2019-12-21T11:17:45  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has joined #opensuse-admin
2019-12-21T12:04:28  *** adrianS <adrianS!~adrian@opensuse/member/AdrianSuSE> has quit IRC (Ping timeout: 260 seconds)
2019-12-21T12:47:48  *** cboltz <cboltz!~cb@opensuse/member/Cboltz> has joined #opensuse-admin
2019-12-21T14:14:29  *** tigerfoot <tigerfoot!~tigerfoot@opensuse/member/Bruno-friedmann> has joined #opensuse-admin
2019-12-21T16:46:57  *** tigerfoot <tigerfoot!~tigerfoot@opensuse/member/Bruno-friedmann> has quit IRC (Quit: Geeko ate the wire!)
2019-12-21T16:48:09  *** tigerfoot <tigerfoot!~tigerfoot@opensuse/member/Bruno-friedmann> has joined #opensuse-admin
2019-12-21T20:50:15  <cboltz> looks like several services served via haproxy (static.o.o, monitor.o.o, kubic.o.o) are currently not reachable :-(
2019-12-21T20:50:33  <cboltz> nmap static.opensuse.org   reports ports 80 and 443 as "closed"...
2019-12-21T20:52:10  <cboltz> kbabioch: any idea what could be wrong?
2019-12-21T20:52:40  <kbabioch> nope, no idea, didn't look into it (yet)
2019-12-21T20:53:39  <cboltz> then please do ;-)
2019-12-21T20:54:28  <cboltz> in case it matters - I've seen some "random" static.o.o outages in the last days which magically fixed themself after a reload
2019-12-21T20:55:10  <cboltz> but today it's a permanent problem which might at least make debugging easier
2019-12-21T20:55:36  <cboltz> ... and now that I wrote that, it just came back
2019-12-21T20:57:43  <kbabioch> hm, interesting ;-/
2019-12-21T20:58:44  <cboltz> that sounds like you didn't even have time to do serious debugging...
2019-12-21T20:59:11  <cboltz> in case it helps - the full nmap scan result (while static.o.o was unreachable) is at https://paste.opensuse.org/702ece5a
2019-12-21T21:01:03  <cboltz> just guessing - is there a firewall/router in front of anna that could have a slightly broken firewall config?
2019-12-21T21:01:32  <cboltz> even guessing more - maybe two of them with a HA setup, and one of them is misconfigured?
2019-12-21T21:01:59  *** srinidhi <srinidhi!~srinidhi@opensuse/member/srinidhi> has quit IRC (Ping timeout: 258 seconds)
2019-12-21T21:02:41  <cboltz> kbabioch: ... and it's down again :-/
2019-12-21T21:03:01  *** ldevulder__ <ldevulder__!~ldevulder@176.167.105.218> has joined #opensuse-admin
2019-12-21T21:06:42  *** ldevulder_ <ldevulder_!~ldevulder@176.167.120.64> has quit IRC (Ping timeout: 260 seconds)
2019-12-21T21:22:00  <kbabioch> now im at the right laptop and can have a look ;-)
2019-12-21T21:22:25  <cboltz> it's still down, so you might even find something
2019-12-21T21:23:56  <cboltz> on the positive side, the wiki (which runs via login2.o.o) is reachable - which also means the problem is likely somehow related to anna
2019-12-21T21:24:41  <kbabioch> anna has the ip, according to keepalive log no failover happened since 9 days (last reboot)
2019-12-21T21:24:52  <kbabioch> the last router that i can still see is a mf core router
2019-12-21T21:25:36  <kbabioch> but for me it works right now
2019-12-21T21:25:40  <kbabioch> is it broken for you?
2019-12-21T21:25:57  <cboltz> it works again
2019-12-21T21:26:41  <cboltz> but I wouldn't be surprised if it breaks again in some minutes
2019-12-21T21:31:45  <cboltz> down again...
2019-12-21T21:35:31  <kbabioch> the only unexpected (and most likely unrelated) thing in the haproxy logs on anna / elsa is that the redmine backend went down in the mean time fora couple of minutes
2019-12-21T21:36:04  <cboltz> indeed, that sounds unrelated
2019-12-21T21:36:29  <kbabioch> narwal{5,6,7} seem to stay available during outage
2019-12-21T21:37:31  <cboltz> right, it must be something between the world and anna - or, even if it sounds unlikely, the firewall on anna
2019-12-21T21:42:50  <kbabioch> i've stopped keepalived on anna now, so that ips fail over to elsa
2019-12-21T21:42:56  <kbabioch> let's wait and see if problem still persists
2019-12-21T21:44:55  <kbabioch> but won't be very responsive throughout the evening/night. will try to have an eye on this, but most likely will only return to look into this tomorrow ...
2019-12-21T21:45:37  <cboltz> yeah, I understand that saturday evening before christmas is the perfect timing :-/
2019-12-21T21:45:49  <cboltz> should we add a note on status.o.o?
2019-12-21T21:47:54  <kbabioch> yeah, we should probably add a note there
2019-12-21T21:48:14  <kbabioch> and say something like "sporadic outages due to reasons not yet fully understood"
2019-12-21T21:48:28  <kbabioch> because right now it works for me (tm) :-)
2019-12-21T21:48:49  <cboltz> sounds like you already have a text ready - can I convince you to paste it to status.o.o? ;-)
2019-12-21T21:51:22  <kbabioch> let me find my credentials :-)
2019-12-21T21:52:16  <kbabioch> do we know what domains exactly are / were affected?
2019-12-21T21:53:37  <cboltz> everything that gets handled via haproxy on anna - which means the list is quite long
2019-12-21T21:53:43  <kbabioch> okay
2019-12-21T21:54:04  <cboltz> things handled by login2.o.o (like the wikis) are not affected - but they look ugly while static.o.o is down
2019-12-21T21:54:44  <kbabioch> and since when is this happening? all day long i presume?
2019-12-21T21:55:11  <cboltz> we got a ticket for html5test today at 15:something
2019-12-21T21:56:06  <cboltz> but I also remember a report on IRC by DimStar yesterday at 13:35
2019-12-21T22:01:04  <kbabioch> https://status.opensuse.org/incidents/202
2019-12-21T22:01:11  <kbabioch> feel free to adjust / update if it is too incorrect
2019-12-21T22:01:15  <kbabioch> i don't know all of the details :-/
2019-12-21T22:03:28  <cboltz> I'm afraid I don't know more details
2019-12-21T22:04:33  <cboltz> one detail is that you marked the "Homepage" as having problems, but it's probably not affected because it's still hosted in Provo
2019-12-21T22:05:18  <cboltz> I'd tend to mark "Data center Nuremberg" as having problems - that's still not completely correct, but slightly better ;-)
2019-12-21T22:08:37  <cboltz> kbabioch: down again...
2019-12-21T22:09:28  <cboltz> traceroute ends at  core-backbone.microfocus.com (5.56.18.210)
2019-12-21T22:11:08  <cboltz> (not sure if that means a lot, traceroute to en.o.o gives me exactly the same route, and the wiki works)
2019-12-21T22:44:06  <cboltz> kbabioch: at the moment, it seems only static.o.o is down - but not other pages served via haproxy like fontinfo.o.o or kubic.o.o
2019-12-21T22:44:40  <cboltz> note that static.o.o uses another IP
2019-12-21T22:45:25  <cboltz> also note that at least kubic.o.o was down before, so switching IPs wouldn't always have helped
2019-12-21T23:12:17  <cboltz> ... and currently static.o.o works, but fontinfo and kubic are down