2020-04-18T00:48:00  -heroes-bot- PROBLEM: MySQL WSREP recv on galera3.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 4.507342 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera3.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-04-18T01:10:55  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 50 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T01:30:55  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=37 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T04:44:55  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 6 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T04:48:13  -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 316 * total waiting locks: 154 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-04-18T04:54:54  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=12 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T04:58:15  -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-04-18T12:32:13  <lcp> kl_eisbaer darix repeating the op question: anyone up to help by being an op or just giving me op so I can handle bridge here (I would probably prefer op because your time is more valuable than this)
2020-04-18T13:48:54  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 166 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T13:58:55  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=15 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T15:37:54  <lcp> kl_eisbaer darix repeating the op question: anyone up to help by being an op or just giving me op so I can handle bridge here (I would probably prefer op because your time is more valuable than this)
2020-04-18T15:58:12  -heroes-bot- PROBLEM: MySQL WSREP recv on galera1.infra.opensuse.org - CRIT wsrep_local_recv_queue_avg = 18.486888 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=galera1.infra.opensuse.org&service=MySQL%20WSREP%20recv
2020-04-18T17:10:37  <cboltz> tuanpembual: I just noticed that we missed a small part when switching over to the new progress VM:
2020-04-18T17:10:58  <cboltz> pillar/id/progress_infra_opensuse.org.sls in the salt repo still says "future progress.o.o" ;-)
2020-04-18T17:11:24  <cboltz> I submitted a fix already: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/381
2020-04-18T17:11:29  <cboltz> please review ;-)
2020-04-18T18:04:55  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 58 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T18:34:54  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=22 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-04-18T18:36:31  -heroes-bot- PROBLEM: HTTP progress on redmine.infra.opensuse.org - connect to address 192.168.47.29 and port 80: Connection refused ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=redmine.infra.opensuse.org&service=HTTP%20progress
2020-04-18T20:25:37  <lcp> cboltz: hm, I still can't access gitlab weirdly enough, any ideas?
2020-04-18T20:25:42  <lcp> it redirects me to opensuse.org automatically
2020-04-18T20:25:49  <lcp> on both chromium and firefox
2020-04-18T20:26:07  <cboltz> strange, especially since I'm just using gitlab ;-)
2020-04-18T20:26:46  <cboltz> to start low-level, what does   host gitlab.infra.opensuse.org   say?
2020-04-18T20:27:06  <cboltz> should be "alias to mickey.i.o.o" and finally 192.168.47.36
2020-04-18T20:30:26  <lcp> that's correct
2020-04-18T20:31:34  * lcp sent a long message:  < https://matrix.org/_matrix/media/r0/download/matrix.org/ZwchwCzPyASCSLPbmqjsphFa >
2020-04-18T20:33:06  <cboltz> interesting[tm] - but I can reproduce it with   curl -v --insecure https://gitlab.infra.opensuse.org/
2020-04-18T20:33:37  <lcp> well then
2020-04-18T20:33:37  <cboltz> try https://gitlab.infra.opensuse.org/infra/salt/ - it gives me/curl a redirect to the gitlab login page
2020-04-18T20:34:09  <lcp> that works
2020-04-18T20:34:10  <lcp> interesting
2020-04-18T20:51:12  <lcp> cboltz: I had a look at the srv record for matrix and https://tools.ietf.org/html/rfc2782 which means we can't really use that srv record method >:T
2020-04-18T20:52:44  <cboltz> what's the problem?
2020-04-18T20:53:08  <lcp> cname
2020-04-18T20:53:26  <lcp> matrix.opensuse.org is an alias of proxy.opensuse.org
2020-04-18T20:53:50  <cboltz> ah
2020-04-18T20:54:34  <cboltz> I'm not a fan of it, but if it helps, we can make matrix.o.o an A+AAAA record
2020-04-18T20:55:14  <cboltz> but before doing that - does cname "only" violate the rfc, or does it cause real-world problems?
2020-04-18T20:56:26  <lcp> rfc is the real world, it defines the way those services connect
2020-04-18T20:56:40  <lcp> which means that this will not work with cname (and that I did verify by trying)
2020-04-18T20:57:19  <cboltz> ok
2020-04-18T20:57:55  <cboltz> refresh my memory - what are the alternatives to the SRV record?
2020-04-18T20:59:36  * cboltz tries to find out if making matrix.o.o A+AAAA or one of the other options is less annoying
2020-04-18T20:59:49  <lcp> .well_known on opensuse.org
2020-04-18T21:00:53  <lcp> https://github.com/matrix-org/synapse/blob/master/docs/delegate.md this basically
2020-04-18T21:01:52  <cboltz> sounds doable - even while it still runs in provo, thanks to the git auto-deploy
2020-04-18T21:13:52  <lcp> cboltz: there is a tiny problem
2020-04-18T21:14:05  <cboltz> why am I not surprised? ;-)
2020-04-18T21:14:23  <cboltz> what exactly?
2020-04-18T21:14:46  <lcp> try curling to opensuse.org/documentation and compare to https://www.opensuse.org/documentation
2020-04-18T21:15:40  <cboltz> there's a redirect to www.o.o - and it looses /documentation :-(
2020-04-18T21:16:02  <cboltz> on the positive side, this redirect is hosted in NBG, so it should be fixable
2020-04-18T21:16:04  <lcp> correct, do you see how that's an issue?
2020-04-18T21:16:17  <cboltz> yes, it will also loose .well-known ;-)
2020-04-18T21:16:41  <lcp> the issue is www it seems
2020-04-18T21:17:19  <cboltz> not really -   curl -v https://opensuse.org/documentation   gives me a redirect to   https://www.opensuse.org/   (without /documentation)
2020-04-18T21:17:39  <lcp> oh, weird
2020-04-18T21:18:10  <lcp> no, that is the issue, it seems the opensuse.org -> www.opensuse.org loses it
2020-04-18T21:18:11  <lcp> every time
2020-04-18T21:18:36  <lcp> that's just annoying, ehhh
2020-04-18T21:18:48  <cboltz> it should be fixable in the haproxy config
2020-04-18T21:18:52  <cboltz> give me some minutes ;-)
2020-04-18T21:35:32  <cboltz> try again ;-)
2020-04-18T21:41:16  <lcp> that's better
2020-04-18T21:42:28  <cboltz> for the records - the difference in haproxy.cfg is   redirect ... location   vs.   redirect ... prefix
2020-04-18T21:43:43  <lcp> ah, righty-o
2020-04-18T21:43:48  <lcp> https://github.com/openSUSE/landing-page/pull/164
2020-04-18T21:44:21  <lcp> as a sidenote, you can use https://federationtester.matrix.org/ for testing the errors with the delegation
2020-04-18T21:44:43  <lcp> just type in `opensuse.org` in the box
2020-04-18T21:46:10  <cboltz> nice ;-)
2020-04-18T21:46:26  <cboltz> let's see if the situation improves in 15 minutes ;-)
2020-04-18T21:47:31  <lcp> eh, I wish I had a vm to setup the account stuff, only a month left >:T
2020-04-18T21:48:17  <cboltz> side question - the documentation for well-known says that you can specify a port
2020-04-18T21:48:27  <lcp> I didn't on purpose
2020-04-18T21:48:45  <lcp> I still want to leave the federation broken for now
2020-04-18T21:49:01  <cboltz> makes sense ;-)
2020-04-18T21:49:18  <lcp> (having it rely on 2 ports also makes scaling work a little bit better)
2020-04-18T21:50:12  <cboltz> OTOH, not needing the highport and only using 443 would be nice ;-)
2020-04-18T21:50:32  <cboltz> but that's a technical detail that you can probably better answer
2020-04-18T21:55:08  <lcp> I would rather not, but in the worst case we could just set it up with 443, sure
2020-04-18T21:55:44  <cboltz> "worst case" sounds like a reason to use the highport ;-)
2020-04-18T21:56:49  <cboltz> (I'm not against the highport, but if you'd have told me that 443 isn't worse, then I'd of course pick that simple solution ;-)
2020-04-18T22:03:29  <cboltz> looks like the www.o.o deployment is on time :-)
2020-04-18T22:43:11  <lcp> cboltz: ah, can you generate a matomo id for chat?