2020-06-22T00:26:29  *** lurchi_ is now known as lurchi__
2020-06-22T02:45:25  -heroes-bot`- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 53 * total waiting locks: 1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-06-22T02:55:25  -heroes-bot`- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=47 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-06-22T02:56:08  *** okurz_ is now known as okurz
2020-06-22T04:41:25  -heroes-bot`- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 4 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-06-22T04:43:14  -heroes-bot`- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 276 * total waiting locks: 134 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-06-22T04:53:14  -heroes-bot`- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=5 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-06-22T07:26:14  <pjessen> any progress with the dns issue on mx[12] ?
2020-06-22T07:41:44  -heroes-bot`- PROBLEM: SSH on metrics.infra.opensuse.org - SSH CRITICAL - OpenSSH_7.9 (protocol 2.0) version mismatch, expected OpenSSH_7.2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=metrics.infra.opensuse.org&service=SSH
2020-06-22T08:50:31  -heroes-bot`- PROBLEM: SSH on gcc-stats.infra.opensuse.org - SSH CRITICAL - OpenSSH_8.3 (protocol 2.0) version mismatch, expected OpenSSH_8.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=gcc-stats.infra.opensuse.org&service=SSH
2020-06-22T11:17:31  <pjessen> is anyone working on mx[12] ?  mails are now beginning to bounce (
2020-06-22T11:17:32  <pjessen> Host or domain name not found. Name service
2020-06-22T11:17:32  <pjessen>     error for name=localhost type=A: Host not found
2020-06-22T11:19:26  <cboltz> great :-/
2020-06-22T11:20:20  * cboltz wonders if we should temporarily switch the o.o MX entries to a subdomain with port 25 closed to prevent the bounces
2020-06-22T11:29:12  <pjessen> cboltz: maybe not a bad idea.
2020-06-22T11:31:49  <cboltz> done, our new MX is now proxy-nue.opensuse.org - which doesn't listen on port 25
2020-06-22T11:31:57  <cboltz> but better more delays than bounces...
2020-06-22T11:32:35  <cboltz> for the records - the original MX entries were mx1.suse.de. and mx2.suse.de., both with prio 42
2020-06-22T11:35:16  <lcp> yikes, what a day
2020-06-22T11:35:27  <lcp> cboltz: mind taking a look https://github.com/openSUSE/landing-page/pull/172
2020-06-22T11:36:04  <cboltz> merged
2020-06-22T11:37:12  <lcp> hopefully that doesn't overload the instance instantly >:D
2020-06-22T11:37:57  <cboltz> that sounds like a problem we want to have ;-)
2020-06-22T11:38:26  <lcp> I don't want to have that problem ;)
2020-06-22T11:38:59  <lcp> I know a few ways how I wouldn't want the instance to fail ;)
2020-06-22T11:39:09  <cboltz> ;-)
2020-06-22T11:40:04  <lcp> as for the IRC bridge, it works way better with lax limit, but I am waiting for some channels to have their mode switch to +S -r
2020-06-22T11:41:29  <cboltz> regarding the landing page - it includes some JS from connect.facebook.net which gets reported as a tracker by firefox
2020-06-22T11:41:44  <cboltz> do you know if we need that script - and if not, can you remove it?
2020-06-22T11:42:14  <lcp> we kinda need it for what the site does
2020-06-22T11:42:35  <lcp> I don't want to yank out that script just to not break other parts of the site
2020-06-22T11:42:58  <cboltz> what exactly would break?
2020-06-22T11:43:11  * cboltz has the script blocked, and didn't notice an obvious problem
2020-06-22T11:43:30  <lcp> I don't remember actually, I looked into it some time ago and decided against it for some reason
2020-06-22T11:49:28  <cboltz> pjessen: looks like mx*.suse.de are improving again - I just submitted a test mail ("telnet mx1.suse.de 25") to my @o.o address
2020-06-22T11:49:51  <cboltz> it's delayed by greylisting on my side, but I'd call that a good sign already
2020-06-22T11:50:02  <cboltz> same result for mx2
2020-06-22T11:57:25  <cboltz> the greylisted mails are still pending (no retry attempt yet), but a new set of test mails went through instantly
2020-06-22T11:57:49  <cboltz> so I think I can switch the MX entries back
2020-06-22T11:57:52  <guillaume_g> Looks like emails servers have problems. I am unable to send emails to openSUSE ML. Here is the error message received: https://paste.opensuse.org/view/raw/20901430
2020-06-22T11:58:36  <cboltz> guillaume_g: when exactly did you get this error? _now_ or > 10 minutes ago?
2020-06-22T12:00:20  <guillaume_g> cboltz: about 10 min ago
2020-06-22T12:01:38  <cboltz> then it's probably already outdated ;-)
2020-06-22T12:02:01  <cboltz> pjessen noticed this problem, and I temporarily "broke" the MX entries for opensuse.org to prevent these bounces
2020-06-22T12:02:22  <guillaume_g> ok :) I need to resend, I guess?
2020-06-22T12:02:25  <cboltz> in the meantime, it looks like mx*.suse.de were fixed
2020-06-22T12:02:38  <cboltz> yes, please resend
2020-06-22T12:03:23  <cboltz> I'll also switch back the MX entries in the hope that mx*.suse.de don't break again (I only see the "outside view" and have no idea who is working on them)
2020-06-22T12:07:56  <cboltz> pjessen: FYI: MX entries are switched back to mx*.suse.de
2020-06-22T12:08:33  <guillaume_g> cboltz: no error message received, but not delivered to ML so far.
2020-06-22T12:09:37  <cboltz> there's probably some delay - until a minute ago, the MX entry pointed to a dead end to avoid more bounces
2020-06-22T12:10:21  <cboltz> it will take some minutes until the DNS update propagates
2020-06-22T12:10:50  <guillaume_g> ack
2020-06-22T12:24:12  <guillaume_g> cboltz: it works again! Thanks
2020-06-22T12:24:52  <klein> we are working on mx{1,2}.suse.de...
2020-06-22T12:34:41  <klein> the relays also have problems, and we are working on them too
2020-06-22T12:35:24  <pjessen> thanks for the update
2020-06-22T12:36:04  <klein> pjessen: since you are outside the SUSE network, can you check if you are able to open http://status.suse.de ?
2020-06-22T12:36:31  <pjessen> klein: server not found
2020-06-22T12:36:31  <klein> we added some info there, but, I (shame on me) forgot to update you guys here
2020-06-22T12:36:48  <klein> yeah... I think we need something "public" for moments like that
2020-06-22T12:36:49  <pjessen> klein: Host status.suse.de not found: 3(NXDOMAIN)
2020-06-22T12:46:21  <lcp> cboltz: hm, so I need to make `curl https://matrix.opensuse.org/_matrix/federation/v1/version -H "Host: matrix.opensuse.org:443"` work, and I kinda wonder how
2020-06-22T12:47:11  <lcp> since 443 isn't accessible from inside the proxy, nginx can't exactly promote it
2020-06-22T12:47:30  <lcp> or maybe I'm not thinking about it properly
2020-06-22T12:57:47  <lcp> cboltz: let's do it differently then, would you do a SRV record like `_matrix._tcp.matrix.opensuse.org. IN SRV 0 0 443 proxy-nue.opensuse.org`
2020-06-22T12:58:14  <lcp> and I will revert the landing_page PR to use the matrix.opensuse.org
2020-06-22T12:58:25  <lcp> it will be a little less work ;)
2020-06-22T13:09:10  <lcp> I updated https://progress.opensuse.org/issues/63463 to reflect that
2020-06-22T13:10:59  <cboltz> lcp: I have to leave, will do the SRV record tonight
2020-06-22T13:12:41  <lcp> thanks!
2020-06-22T13:57:37  <klein> MX are working, relay seems to work too
2020-06-22T13:57:52  <klein> only imap.suse.de is having problems to contact mx/relay servers, working on it
2020-06-22T14:08:37  <pjessen> cool
2020-06-22T15:29:31  <witek_> hello, could someone please grant me access to Hero network?
2020-06-22T15:29:46  <witek_> here's the ticket: https://progress.opensuse.org/issues/68251
2020-06-22T15:41:31  *** lurchi_ is now known as lurchi__
2020-06-22T17:55:19  *** lurchi__ is now known as lurchi_
2020-06-22T18:21:16  <cboltz> lcp: does the SRV record for matrix really need to point to proxy-nue.o.o? IMHO matrix.o.o would make more sense
2020-06-22T18:21:54  <lcp> cname is why
2020-06-22T18:22:18  <lcp> matrix.o.o has cname of proxy-nue, so it has to be like that
2020-06-22T18:27:53  <cboltz> ah, right - now I remember that matrix doesn't like cnames too much...
2020-06-22T18:27:57  <cboltz> changed
2020-06-22T18:29:14  <cboltz> is there something else left in the ticket? (If not, feel free to close it)
2020-06-22T18:29:35  <lcp> yeah, pulling the PR in landing-page
2020-06-22T18:29:43  <lcp> and testing if stuff works >:D
2020-06-22T18:31:03  <cboltz> PR for matrix merged
2020-06-22T18:31:41  <lcp> I will check if stuff works then
2020-06-22T18:32:08  <lcp> in aprox half an hour
2020-06-22T18:32:24  <cboltz> facebook JS removal also merged - one spy less on www.o.o :-)
2020-06-22T18:41:40  <cboltz> klein: are the mx servers completely fixed again? If so, please update status.o.o ;-)  (and please include a note about delayed vs. bounced mails)
2020-06-22T18:57:37  *** lurchi_ is now known as lurchi__
2020-06-22T18:58:12  *** lurchi__ is now known as lurchi_
2020-06-22T19:06:51  <lcp> cboltz: hm, are you sure you set the SRV record  correctly? it returns proper values when using `dig -t srv _matrix._tcp.opensuse.org` but not `dig -t srv _matrix._tcp.matrix.opensuse.org`, which is what is needed
2020-06-22T19:08:49  <lcp> basically in this case we are using double delegation, because of proxy not accepting ports in host headers
2020-06-22T19:13:14  <cboltz> good point, currently we only have _matrix._tcp.o.o
2020-06-22T19:15:08  <cboltz> _matrix._tcp.matrix.o.o added
2020-06-22T19:15:28  <lcp> essentially this is a very neat trick to avoid using port 8448 with a proxy that doesn't accept port numbers in host headers ;)
2020-06-22T19:15:51  <lcp> idk if the proxy not accepting those is a problem anywhere else, but it sure is here ;)
2020-06-22T19:16:34  <cboltz> most things we run through haproxy are boring https pages that just need 443 ;-)
2020-06-22T19:16:57  <cboltz> (I'll ignore details like routing mysql through haproxy)
2020-06-22T19:16:58  <lcp> Host: matrix.opensuse.org:443 also won't work
2020-06-22T19:17:42  <lcp> technically that's breaking a few RFCs
2020-06-22T19:18:04  <lcp> «The Host request-header field specifies the Internet host and port number of the resource being requested, as obtained from the original URI given by the user or referring resource» - RFC 2616
2020-06-22T19:18:49  <lcp> conveniently for us, most software strips those before it reaches the proxy
2020-06-22T19:24:44  <cboltz> that, and most users are too lazy to type   :443   in their browser ;-)
2020-06-22T19:27:02  <lcp> cboltz: `Get matrix://matrix.opensuse.org/_matrix/federation/v1/version: dial tcp: lookup proxy-nue.opensuse.org.opensuse.org on 8.8.8.8:53: no such host`
2020-06-22T19:27:31  <lcp> so I assume it should be `proxy-nue` and not `proxy-nue.opensuse.org` in the srv
2020-06-22T19:27:59  <lcp> note there are too many `opensuse.org`s at the end there
2020-06-22T19:29:23  <cboltz> good point (literally) - I forgot the trailing dot
2020-06-22T19:29:41  <cboltz> fixed
2020-06-22T19:29:54  <lcp> righty-o, that would be it
2020-06-22T19:32:09  <lcp> yup, federation works now
2020-06-22T19:33:06  <cboltz> :-)
2020-06-22T20:33:17  *** Martchus_ is now known as Martchus
2020-06-22T21:08:35  *** lurchi_ is now known as lurchi__
2020-06-22T21:11:58  *** LCP[m]11 is now known as lcp
2020-06-22T21:27:42  *** lurchi__ is now known as lurchi_