2020-04-27T05:27:42 -heroes-bot- PROBLEM: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB mb_opensuse2 (host:mirrordb2) 76834344 and 1 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby 2020-04-27T05:37:42 -heroes-bot- RECOVERY: PostgreSQL standby on mirrordb1.infra.opensuse.org - POSTGRES_HOT_STANDBY_DELAY OK: DB mb_opensuse2 (host:mirrordb2) 3368 and 1 seconds ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PostgreSQL%20standby 2020-04-27T08:12:19 Hi all! Recently we started to get e-mails about overdue from progress. We have figured out that those e-mails are sent from the old progress instance. Could you please suggest how to solve this, as hostname I get in the e-mail is not resolvable (http://redmine.infra.opensuse.org/), so I cannot login to old progress instance to disable e-mail notifications 2020-04-27T11:00:51 riafarov: should be fixed - I stopped redmine on the old server when we switched over, but for some reason (no idea way) it was running again. Stopped again. 2020-04-27T11:01:17 cboltz: Thanks a lot for your help! Very appreciated! 2020-04-27T11:05:32 cboltz: you didnt turn off the service and someone rebooted it for kernel updates? 2020-04-27T11:05:35 just a guess 2020-04-27T11:06:39 you are right with the first half (done now, I feel old doing "insserv -r" - especially because I didn't have to look it up ;-) 2020-04-27T11:06:51 the first half is unlikely - uptime is 132 days 2020-04-27T11:07:01 ok 2020-04-27T11:07:12 well maybe we can just destroy the VM?:P 2020-04-27T11:08:22 Lars already asked (IIRC last week) - IMHO we can shut it down, but keep the image for some weeks just in case we still need something from it 2020-04-27T11:51:13 cboltz: I will need temporary ssh access on fedora-sso through sso.o.o, how possible does that sound 2020-04-27T11:52:09 lcp: why "through sso.o.o" ?:) 2020-04-27T11:52:12 dont you have vpn 2020-04-27T11:53:02 did you see this already? :) https://smallstep.com/blog/use-ssh-certificates/ 2020-04-27T11:53:08 while we talk about ssh 2020-04-27T11:54:36 interesting 2020-04-27T11:55:06 speaking of interesting security articles, have you seen this? https://latacora.micro.blog/2019/07/16/the-pgp-problem.html 2020-04-27T11:56:38 also for that matter, isn't freeipa supposed to deal with handing out ssh keys where necessary 2020-04-27T11:59:26 lcp: you dont want to have ssh keys distributed :) 2020-04-27T11:59:37 with ssh certs you only get temporary access 2020-04-27T12:00:41 lcp: I have a package for step-ca ;) 2020-04-27T12:00:48 if you have openid connect working 2020-04-27T12:00:51 we could set it up :P 2020-04-27T12:03:37 yeah, I think we can figure it out later, right now I would like to deal with sso working, which requires a temporary access to ssh 2020-04-27T12:56:24 yup, that works 2020-04-27T12:57:34 what the?! 2020-04-27T13:03:08 delays between irc and matrix :/ 2020-04-27T13:43:52 lcp, Eighth_Doctor the forum move may happen sooner (well from a data dump perspective) than planned, so do you have a realistic time as to if/when SSO will work? 2020-04-27T13:45:18 we just want to advise users how long the forum will be read-only 2020-04-27T13:47:38 how soon are we talking 2020-04-27T13:49:04 lcp, could be this week... 2020-04-27T13:50:45 lcp, I'm still trying to get details, but need the lead from you guys first if it's going to be some time, it is what it is... 2020-04-27T13:52:05 well, we are mostly ready, ignoring some kinks here and there, but this move mostly depends on moving every existing application, which makes this problematic 2020-04-27T13:53:52 lcp, ok, I'll come back once I know some more and we can take it from there :) Thanks :) 2020-04-27T15:35:06 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 58 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-27T15:55:06 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=47 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-27T16:19:06 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 66 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-27T16:44:12 since nobody complained, I'll switch www.o.o to Nuremberg now 2020-04-27T16:44:25 for the records - the old DNS entry is www A 130.57.66.6 2020-04-27T16:45:07 and the new one is CNAME proxy.opensuse.org. 2020-04-27T16:45:30 I'll also delete www-new.o.o - we no longer need it 2020-04-27T16:49:18 -heroes-bot- PROBLEM: DNS on chip.infra.opensuse.org - DNS CRITICAL - expected 130.57.66.6 but got 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=chip.infra.opensuse.org&service=DNS 2020-04-27T16:50:57 -heroes-bot- PROBLEM: DNS on freeipa.infra.opensuse.org - DNS CRITICAL - expected 130.57.66.6 but got 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=freeipa.infra.opensuse.org&service=DNS 2020-04-27T16:51:56 nice, looks like we have a DNS check that just found out that I switched over www.o.o to Nuremberg ;-) 2020-04-27T16:52:09 kl_eisbaer: can you please update the expected IP in the monitoring? 2020-04-27T16:53:48 cboltz: Will do. FYI: I'm currently transferring everything from old icinga to new icinga2 2020-04-27T16:54:32 -heroes-bot- PROBLEM: DNS on nue-ns1.infra.opensuse.org - DNS CRITICAL - expected 130.57.66.6 but got 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nue-ns1.infra.opensuse.org&service=DNS 2020-04-27T16:54:49 I have a WTF moment here: ~> host www.opensuse.org 2020-04-27T16:54:50 www.opensuse.org has address 130.57.66.6 2020-04-27T16:54:50 www.opensuse.org is an alias for proxy.opensuse.org. 2020-04-27T16:54:50 proxy.opensuse.org is an alias for proxy-nue.opensuse.org. 2020-04-27T16:55:17 what, how 2020-04-27T16:55:51 I deleted the A record (actually _had to_ because freeipa doesn't allow to have cname and A for the same domain) 2020-04-27T16:55:52 oh yeah, this is actually the output 2020-04-27T16:56:07 -heroes-bot- PROBLEM: DNS on nue-ns2.infra.opensuse.org - DNS CRITICAL - expected 130.57.66.6 but got 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nue-ns2.infra.opensuse.org&service=DNS 2020-04-27T16:56:20 At least my local DNS looks currently confused 2020-04-27T16:56:27 maybe you have some DNS cache playing funny games with you? 2020-04-27T16:56:28 yeah, same 2020-04-27T16:57:00 nope, flushing the dns, still the same 2020-04-27T16:57:02 Heya: we come much closer to a dual-homed opensuse.org domain :-) 2020-04-27T16:57:03 I restarted my local unbound after the change, and host www.o.o looks like expected (cname proxy etc.) 2020-04-27T16:57:11 lol 2020-04-27T16:57:25 -heroes-bot- PROBLEM: DNS on qsc-ns3.infra.opensuse.org - DNS CRITICAL - expected 130.57.66.6 but got 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=qsc-ns3.infra.opensuse.org&service=DNS 2020-04-27T16:57:28 9.9.9.9 is it 2020-04-27T16:57:49 ^^ this one still has the old A entry 2020-04-27T16:58:06 Google seems to be fine 2020-04-27T16:58:11 1.1.1.1 is also answering wrong here 2020-04-27T16:58:40 funny, for me 9.9.9.9 only gives me the cname, while 8.8.8.8 gives me (only) the old A record 2020-04-27T16:59:02 but as usual with DNS - let's wait for all the caches ;-) 2020-04-27T16:59:05 at least our 3 own servers answer correctly incorrect (see monitoring ;-) 2020-04-27T17:00:56 -heroes-bot- RECOVERY: DNS on freeipa.infra.opensuse.org - DNS OK: 0.014 seconds response time. www.opensuse.org returns 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=freeipa.infra.opensuse.org&service=DNS 2020-04-27T17:01:15 -heroes-bot- RECOVERY: DNS on chip.infra.opensuse.org - DNS OK: 0.014 seconds response time. www.opensuse.org returns 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=chip.infra.opensuse.org&service=DNS 2020-04-27T17:01:17 -heroes-bot- RECOVERY: DNS on nue-ns1.infra.opensuse.org - DNS OK: 0.010 seconds response time. www.opensuse.org returns 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nue-ns1.infra.opensuse.org&service=DNS 2020-04-27T17:01:19 -heroes-bot- RECOVERY: DNS on nue-ns2.infra.opensuse.org - DNS OK: 0.013 seconds response time. www.opensuse.org returns 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=nue-ns2.infra.opensuse.org&service=DNS 2020-04-27T17:01:21 -heroes-bot- RECOVERY: DNS on qsc-ns3.infra.opensuse.org - DNS OK: 0.011 seconds response time. www.opensuse.org returns 195.135.221.140,2620:113:80c0:8::16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=qsc-ns3.infra.opensuse.org&service=DNS 2020-04-27T17:13:10 kl_eisbaer, lcp: can you please have a look at https://paste.opensuse.org/44b2176b 2020-04-27T17:13:27 I'd like to send that out to the heroes and project ML 2020-04-27T17:16:19 seems good 2020-04-27T17:16:51 I am really not sure about what the timeline for sso is gonna be, since we have quite a bunch of stuff going on at the same time 2020-04-27T17:20:03 all the balls are in the air... 2020-04-27T17:34:23 as if we needed any more stressful stuff going on >:D 2020-04-27T17:35:11 BTW: did I tell you that we need to renumber some machines IPv6 addresses, still? 2020-04-27T17:35:16 :-P 2020-04-27T17:35:53 Everything starting with 2620:* needs to get a new address.... 2020-04-27T17:36:01 IIRC yes 2020-04-27T17:36:16 the good thing is that this "only" affects the machines with public IPs, so the number isn't too big 2020-04-27T17:36:56 one detail (not sure if I mentioned it before) - please make sure to get the reverse DNS setup first so that mail delivery doesn't run into problems 2020-04-27T17:38:10 kl_eisbaer: The stats api of the videobridge is running 2020-04-27T17:38:35 but only locally so far. The port is not exposed yet 2020-04-27T18:04:41 cboltz: if you want we can quickly assign a 2620 IP to all infra.o.o addresses and switch off ipv4 2020-04-27T18:04:44 ;) 2020-04-27T18:04:48 *runs* 2020-04-27T18:06:54 genius 2020-04-27T18:07:46 lcp: we were thinking about going ipv6 only internally a few times. 2020-04-27T18:08:40 Getting rid of IPv4 is like getting rid of ISDN or python2 ;) 2020-04-27T18:09:24 lethliel: well telekom bullied out everyone on ISDN and moved them to voip 2020-04-27T18:09:27 so that part is done 2020-04-27T18:09:37 and python2 ... dimstar is already looking forward to it 2020-04-27T18:09:55 well, we don't rely on python2 for most things by now 2020-04-27T18:10:12 I guess it's only mb now 2020-04-27T18:10:16 we got rid of rawdog 2020-04-27T18:10:24 lcp: actually MB is ported ;) 2020-04-27T18:10:47 lcp: it is even python3 only now! 2020-04-27T18:10:53 death to python2 2020-04-27T18:11:00 darix: But it was a long way 2020-04-27T18:11:14 lethliel: sure. but it is happening 2020-04-27T18:11:21 if you turn off ipv4 access, I will not be able to access anything 2020-04-27T18:11:27 lethliel: image you could rip out all the python 2 compat out of osc ;))) 2020-04-27T18:11:30 I don't have ipv4 support here 😭 2020-04-27T18:11:37 darix: hehe :-) 2020-04-27T18:11:38 err, ipv6 support 😭 2020-04-27T18:11:58 darix: well, it "should" support python3 2020-04-27T18:12:03 Eighth_Doctor: well you can do a tunnel over ipv4 that then runs ipv6 internally 2020-04-27T18:12:27 lcp: no for mirrorbrain we went with. "why keep supporting a dead horse" :P 2020-04-27T18:12:31 and made it py3 only 2020-04-27T18:12:32 :P 2020-04-27T18:12:42 But I mean for ISDN. As I started to work for Telekom in 2000 they stated that ISDN is dead soon. I left the telekom 16 years later and we still had some ISDN lines left 2020-04-27T18:13:47 I hope we don't have any hidden python2 anywhere 2020-04-27T18:13:54 that would be a nasty surprise with Leap 16 when that comes 2020-04-27T18:14:06 lcp: we probably have. 2020-04-27T18:14:41 we should enforce versioned shebangs at package builds for python stuff :) 2020-04-27T18:15:05 a lot of things in Fedora got discovered and fixed when we did that and broke package builds using /usr/bin/python :) 2020-04-27T18:15:05 right 2020-04-27T18:15:38 Eighth_Doctor: last time I asked dimstar ... only 398 were left that needed python ;) 2020-04-27T18:16:56 the innovators with their dependency on rust left with python2? impossible 2020-04-27T18:17:09 I don't believe it for a second 2020-04-27T18:19:22 mozilla tensorflow and 396 other packages 2020-04-27T18:21:43 not to mention chromium 2020-04-27T19:04:34 I am off for today 2020-04-27T19:13:57 lethliel: no worries: I can check the status of the videobridge via a local command. That's absolute ok :-) 2020-04-27T19:42:25 cboltz: would you mind redirecting www.o.o/searchPage to search.o.o? 2020-04-27T19:48:56 should work now 2020-04-27T19:51:21 I think it's time to add a favicon.ico and a robots.txt to git - these two are the most "popular" entries in the error log ;-) 2020-04-27T19:51:25 can you please do that? 2020-04-27T19:53:11 cboltz: or just redirect the favicon to static.opensuse.org? 2020-04-27T19:53:28 yes, that sounds like a better solution tbh 2020-04-27T19:53:40 central place for static :P 2020-04-27T19:54:01 about robots.txt, I have no idea what robots.txt should contain, so I will look that up 2020-04-27T19:54:15 the robots.txt you can put into haproxy too 2020-04-27T19:54:37 at least for all the generic cases which just have "allow / for all" 2020-04-27T19:54:39 that will need some changes in the haproxy config - currently I use the same backend for "all static pages", so simply adding a redirect would also trigger a redirect from static.o.o/favicon to static.o.o/favicon 2020-04-27T19:54:55 so I'll probably have to make a separate backend only for static.o.o 2020-04-27T19:55:02 cboltz: you dont 2020-04-27T19:55:28 acl favicon path /favicon.ico 2020-04-27T19:56:09 redirect location https://static.opensuse.org.org/favicon.ico code 301 if favicon !static 2020-04-27T19:57:16 right now, several domains (shop, static.o.o, html5test) all get mapped to is_static, which makes things slightly more interesting 2020-04-27T19:57:29 but then, I could do a acl in the backend that only matches static.o.o 2020-04-27T19:58:27 I would do that as early as possible 2020-04-27T19:58:49 so make 2 hdr(host) acls 2020-04-27T19:58:53 one for the "catchall" 2020-04-27T19:58:56 one for static 2020-04-27T19:59:01 and then 2020-04-27T19:59:10 use_backend narwal if catchall || static 2020-04-27T19:59:24 right, makes sense 2020-04-27T20:00:18 any idea for a good name for "catchall"? The obvious choice would be "static", but... ;-) 2020-04-27T20:00:47 (hmm, "static_o_o" for static.o.o, and just "static" for everything else?) 2020-04-27T20:00:52 is_staticpages 2020-04-27T20:00:56 is_staticoo 2020-04-27T20:01:19 sounds good, I'll use that 2020-04-27T20:22:30 *** lilly[m]1 is now known as lilly[m]2 2020-04-27T20:22:31 *** lilly[m]2 is now known as lilly[m]4 2020-04-27T20:22:31 *** lilly[m]4 is now known as riku[m] 2020-04-27T20:26:37 done, all domains hosted on narwal should now use/redirect to the favicon from static.o.o 2020-04-27T20:27:04 cboltz: if you added my rule ... it is basically all domains :P 2020-04-27T20:27:49 well, I'm not sure if we really want it for all domains - let's start with "just" the static pages ;-) 2020-04-27T20:31:30 :P 2020-04-27T22:54:32 lcp: www.opensuse.org/build/fonts/glyphicons-halflings-regular.eot goes 404, but is referenced in https://www.opensuse.org/build/css/openSUSE.min.css 2020-04-27T22:54:38 can you please have a look at that? 2020-04-27T22:55:57 eh, we are using gstatic there 2020-04-27T22:57:37 I can only tell you that the error_log has a few requests for that font ;-) 2020-04-27T22:58:46 and we also have facebook sdk, even if we aren't using it 2020-04-27T22:58:47 why 2020-04-27T22:59:14 if you are sure that we don't use it, just remove it 2020-04-27T23:00:02 that's not that simple, since that's compiled js >:D 2020-04-27T23:01:09 there's also the un-compiled js in the repo - assets/js/ 2020-04-27T23:02:04 and the README.md explains how to compile it 2020-04-27T23:02:21 (never tried that ;-) 2020-04-27T23:25:08 *** Skull[m]1 is now known as Skull3116[m] 2020-04-27T23:25:08 *** Skull3116[m] is now known as Skull[m]2 2020-04-27T23:25:09 *** Skull[m]2 is now known as Skull[m4