2020-04-28T04:42:05 -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 260 * total waiting locks: 126 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks 2020-04-28T04:45:06 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 4 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-28T04:52:05 -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks 2020-04-28T04:55:06 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-28T05:41:43 *** lindsey[m] is now known as lindsey[m]4 2020-04-28T05:41:44 *** lindsey[m]4 is now known as lindsey[m] 2020-04-28T05:41:44 *** lindsey[m] is now known as lindsey[m]6 2020-04-28T10:59:07 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 56 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-28T11:09:06 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=40 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-28T11:33:07 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 56 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-04-28T16:38:47 cboltz: can we get a www.o.o/openidlegacy to sso.o.o/openidlegacy redirect? 2020-04-28T16:39:27 this is preparation for www.o.o/openid obviously 2020-04-28T16:39:58 redirect or proxying? 2020-04-28T16:40:05 redirect 2020-04-28T16:40:27 we will also need a *.sso.o.o certificate generated somehow 2020-04-28T16:43:02 https://sso.opensuse.org/openidlegacy says 404, is this intentional? 2020-04-28T16:44:18 (nevertheless, the redirect is there now) 2020-04-28T17:23:19 cboltz, lcp: do you know if we use openSUSE services that depend on openID ? 2020-04-28T17:23:57 kl_eisbaer: I wrote which ones do in my email 2020-04-28T17:23:59 I'm currently thinking if there is - at the moment - any application using it for real 2020-04-28T17:24:19 Jenkins and openQA 2020-04-28T17:24:23 lcp: oh, sorry. missed that :-) 2020-04-28T17:24:38 I don't think we have much choice with openQA tbh 2020-04-28T17:24:53 also paste.o.o, but I also know that the login is broken ("The URI you submitted has disallowed characters.") 2020-04-28T17:25:10 cboltz: that's openid php library 2020-04-28T17:25:15 also in my email 2020-04-28T17:25:26 slash is a disallowed character 2020-04-28T17:25:48 www.opensuse.org/openid/user contains slashes 2020-04-28T17:26:25 hm 2020-04-28T17:26:26 sadly this is the kind of stuff in which I had to become an expert ;) 2020-04-28T17:26:49 I'm asking because someone from our IT team asked me if openSUSE still needs the openID provider from MF 2020-04-28T17:27:02 well, they internally also do need that 2020-04-28T17:27:13 I wonder how they want to fix internal openQA 2020-04-28T17:27:45 lcp: is there no way to use other authentication mechanisms for Jenkins and openQA ? 2020-04-28T17:28:12 not as far as I'm aware, but let me check 2020-04-28T17:28:47 Jenkins seems to have some auth plugins 2020-04-28T17:28:52 https://plugins.jenkins.io/ui/search?query=auth 2020-04-28T17:28:55 ah, it does 2020-04-28T17:29:04 maybe okurz[m] can tell us something about openQA :-) 2020-04-28T17:29:25 in any case, it uses openid for now, so it will continue to work 2020-04-28T17:30:04 currently openQA only supports openid and fake (which is development only) 2020-04-28T17:31:28 I am really hoping SUSE-IT put some thought into that stuff before deciding on what the did, because this doesn't really sound reassuring tbh ;) 2020-04-28T17:31:37 * I am really hoping SUSE-IT put some thought into that stuff before deciding on what they did, because this doesn't really sound reassuring tbh ;) 2020-04-28T17:32:57 lcp: you don't want to get a honest answer to this. Really. No. 2020-04-28T17:34:49 yeah, I was expecting as much 2020-04-28T17:35:35 "Hey! Looks at least as there is still 1.5 weeks left to implement missing features in your applications..." 2020-04-28T17:35:58 kl_eisbaer LCP already mentioned that openQA should be not problem with openID, right? 2020-04-28T17:36:12 I can tell you from own experience that last-minute development is a well-proven method ;-) 2020-04-28T17:36:18 on openSUSE side, that's not an issue 2020-04-28T17:36:29 okurz[m]: well: the question is more if openQA would have a problem *without* openID ... 2020-04-28T17:36:31 and 1.5 weeks does _not_ qualify as last-minute 2020-04-28T17:36:32 SUSE doesn't have an openID solution it seems 2020-04-28T17:37:04 we did report the issue with OBS in February, and expected to deploy accounts half a year later, before we learned we have time until may 2020-04-28T17:37:50 which well, sucks, but is way better than not actually considering everything that already exists months in advance 2020-04-28T17:38:29 lcp: did I tell you that I sometimes miss my "admin power", I had a few years ago? 2020-04-28T17:38:55 lcp: but I have to admit that today I'm also totally happy to see how others do this job ... 2020-04-28T17:39:11 ...not. 2020-04-28T17:39:30 Yes, openqa.suse.de as well as openqa.opensuse.org would have problems without openID and I would be annoyed if someone just shuts off the openID service without informing us. The admin of the openID server should see all the openQA servers using it 2020-04-28T17:40:24 kl_eisbaer: at some point I will just have to ask the board to help us with convincing SUSE it would be a good idea to have all of the deployment stuff (aka. salt profiles) public instead for the openSUSE services ;) 2020-04-28T17:40:31 that will be a fun thing to do 2020-04-28T17:40:33 okurz[m]: let me bring this up to some people tomorrow. Let's see, how they like to handle that. 2020-04-28T17:41:19 Appreciated. So I am not sure I got the message: is it planned to not have openID then? 2020-04-28T17:41:26 lcp: you are talking about the salt profiles in gitlab.i.o.o ? 2020-04-28T17:41:47 lcp: IMHO there should not be a big problem in making them public. 2020-04-28T17:42:25 nono, making them public for heroes, not for everyone 2020-04-28T17:42:28 the deployments of stuff like software-o-o and openSUSE BS to be in the salt repo ;) 2020-04-28T17:42:58 okurz[m]: that's what I heard today. But there will be a meeting tomorrow... 2020-04-28T17:43:35 lcp: I'm really not sure if software.o.o is deployed by Salt. It's IMHO just the package (from OBS) and the connection to the database. But I can check this. 2020-04-28T17:44:14 lcp: for openSUSE BS, you are talking to the one who can decide to release it.... 2020-04-28T17:44:46 lcp: an my team already discussed this a few month ago.... 2020-04-28T17:44:48 well, considering alex moved over to salt team, it might be >:D 2020-04-28T17:45:48 lcp: releasing the salt profiles to the public should IMHO not be a big problem - but I'm not sure if they would be helpful for someone ? 2020-04-28T17:45:55 nice, that would be great 2020-04-28T17:46:03 the question is just: where? 2020-04-28T17:46:16 should we put a copy in gitlab.i.o.o ? 2020-04-28T17:47:28 yes, please 2020-04-28T17:47:30 I guess some stuff might be interesting there - like the "haproxy config generator" 2020-04-28T17:48:16 I know a guy that has been trying to get some of the info about this stuff from OBS team and never could 2020-04-28T17:48:16 that sounds *very* interesting - even if I'm more familiar with haproxy in the meantime, I always "enjoy" to edit a 1000 line config file... 2020-04-28T17:48:21 this would be amazing resource 2020-04-28T17:48:33 ok. Would it be ok if we start with some small pieces? 2020-04-28T17:48:47 of course 2020-04-28T17:49:03 Otherwise we would need to wait until all this "carve out stuff" is over 2020-04-28T17:49:36 ok. Let me chat with Darix and Rudi than, how we can provide some interesting bits and pieces. 2020-04-28T17:51:23 I'm just happy that LCP and I have managed to start pulling this together 2020-04-28T17:51:44 we started working on this back in early February, and now we have a system that (mostly) works 2020-04-28T17:52:01 though I really hate vBulletin now 2020-04-28T17:52:11 ;-) 2020-04-28T17:52:18 and we mostly know how to fix stuff that's broken, that's probably more important 2020-04-28T17:52:25 yep 2020-04-28T17:52:38 both of us are increasingly getting familiar with how the stack works, which is good 2020-04-28T17:52:51 BTW: lcp: anything missing on the postgresql side? 2020-04-28T17:52:57 and there are experts we can reach out to that are happy to help us out 2020-04-28T17:53:00 I thought you have access to this small cluster anyway? 2020-04-28T17:53:28 right, I need to change from internal to external psql server 2020-04-28T17:53:52 we had a need to debug database entries due to reasons, so that did make sense for initial deployment 2020-04-28T17:54:24 but yeah, I can now switch over 2020-04-28T17:55:04 JFYI: there is a pgbouncer running on anna/elsa - having a pg_hba.conf 2020-04-28T17:55:28 Plus the standard pg_hba.conf on the PGSQL machines - but this one normally just allows anna/elsa 2020-04-28T17:55:35 ah, alright 2020-04-28T17:55:50 yup 2020-04-28T17:56:20 the main point (I also forgot initially): don't forget to apply the pgbouncer.access.sql schema to a newly created database :-) 2020-04-28T17:56:38 noted 2020-04-28T17:56:51 The schema is in /var/lib/pgsql/pgbouncer.access.sql on the master node: mirrordb1 2020-04-28T17:57:07 hm: maybe this should be written down somewhere ... :-) 2020-04-28T17:57:17 probably :P 2020-04-28T17:57:52 we should also write how to generate metadata for SAML2 and OpenID Connect authentication with clients, so everyone knows 2020-04-28T17:58:09 If you forget to apply the schema, the user on your host can not log in, as pgbouncer does not know him. Which is really nothing easy to figure out if you do not work that often with such stuff 2020-04-28T17:58:36 jip 2020-04-28T17:58:37 ah, that makes sense 2020-04-28T17:58:50 looks like we all won some "documentation, please" topics :-) 2020-04-28T17:59:06 Damn: I just missed cboltz this time 2020-04-28T17:59:07 this metadata needs to be sent out to the clients that want to use those methods, so it's really really important 2020-04-28T17:59:27 lcp: you mean the services, right? 2020-04-28T17:59:37 yes 2020-04-28T17:59:52 well, what is a service if not a client of a provider >;D 2020-04-28T17:59:55 ok - but meanwhile they should be under our control anyway :-D 2020-04-28T18:00:30 but, they don't have to, which is nice, because you can create a dynamic client with subset of permissions for people outside of infra 2020-04-28T18:00:47 makes it possible to have openSUSE login, without needing to host stuff internally 2020-04-28T18:00:57 you mean like paste.o.o - I understand 2020-04-28T18:01:00 and we might use that for example for release-monitoring.org integration 2020-04-28T18:01:08 kl_eisbaer: If i get icinga warnings of localhost load the monitoring host itself is meant, right? 2020-04-28T18:01:19 that would be nice, because yahoo and fedora logins are meh 2020-04-28T18:01:21 lethliel: ups, yes, sorry 2020-04-28T18:01:32 you mean from the old monitoring, right? 2020-04-28T18:01:36 yes 2020-04-28T18:01:56 That machine is currently a bit stressed, as I run the old and new monitoring in parallel for now, until I have all checks migrated 2020-04-28T18:02:20 I was just puzzled who localhost is... 2020-04-28T18:02:32 lethliel: hey: localhost == 127.0.0.1 2020-04-28T18:02:41 so normally it's your machine :-D 2020-04-28T18:02:46 ;-) 2020-04-28T18:03:19 kl_eisbaer: I'll happily _read_ that documentation, because both postgresql and all that auth stuff are topics where I can (have to?) learn a lot ;-) 2020-04-28T18:03:27 * cboltz switches to the AppArmor meeting 2020-04-28T18:04:08 Anyone planning to use meet.o.o on friday? Because I would like to swich to a new version... 2020-04-28T18:04:26 lethliel: fixed. short term solution: just comment out localhost. Long term solution: never install a package update any more ;-) 2020-04-28T18:04:58 lethliel: just schedule a downtime on https://status.opensuse.org/ 2020-04-28T18:05:38 Ok. :-) Never update anymore sounds about right. 2020-04-28T18:05:45 hm: and maybe create a component before... Gimme a sec 2020-04-28T18:06:02 lethliel: how do you like your meet.opensuse.org being listed on status.opensuse.org ? 2020-04-28T18:06:14 As "Video conferencing" ? 2020-04-28T18:06:21 kl_eisbaer: Wait. How do you do this? 2020-04-28T18:06:24 or "Jitsi server?" 2020-04-28T18:06:30 lethliel: https://status.opensuse.org/dashboard/components 2020-04-28T18:06:48 Just log in with the "special" credentials I send you years ago :-) 2020-04-28T18:07:23 just tell me when you are logged in 2020-04-28T18:07:44 must find the creds first. 2020-04-28T18:08:14 Ha! reminder to self: don't shut down your workstation, if you have a cron job running that feeds the metrics on status.opensuse.org :-) 2020-04-28T18:21:55 cboltz: I might have a fun quest for you, rerouting all *.sso.opensuse.org connections to fedora-sso server 2020-04-28T18:23:00 also I assume we will need a new cert for this too 2020-04-28T18:25:21 the cert needs to be done by someone @SUSE 2020-04-28T18:25:57 and for the wildcard domain - well, haproxy supports regex matches which makes this quite easy ;-) 2020-04-28T18:27:28 do you have a specific set of characters that should be allowed for the * like [a-z0-9_-] or should I just allow [^.]*\.sso\.opensuse\.org? 2020-04-28T18:34:37 cboltz: the match should be everything we accept for usernames 2020-04-28T18:34:59 yes, of course 2020-04-28T18:35:05 so - what do we accept for usernames? ;-) 2020-04-28T18:35:24 * cboltz hopes "." isn't allowed, because that might cause certificate fun 2020-04-28T18:38:26 that's freeipa that decided that 2020-04-28T18:38:41 uh, although novell might have been very lenient about this 2020-04-28T18:39:19 well, novell sure did allow dots >:T 2020-04-28T18:40:14 certificate fun then, at least for those users with dots 2020-04-28T18:41:03 we will have to see how many users this affects then 2020-04-28T18:41:19 in the worst case, we might just replace with another character and make sure there isn't an overlap 2020-04-28T18:42:05 I will actually see if freeipa doesn't restrict some characters that might have been in the usernames tho 2020-04-28T18:44:48 well, posix is clearly against us here 2020-04-28T18:46:13 we will have to cut out quite a few characters, but we don't have to remove dots, since that's apparently supported under posix 2020-04-28T18:47:35 posix specifies [A-Za-z0-9._-], novell/mf specifies [A-Za-z0-9~!@#$%^*()-_=[]{}|;:<>,.?. ] 2020-04-28T18:48:58 Conan Kudo: this will be fun ;) 2020-04-28T18:49:06 sounds like we'll have some fun ;-) (and I'm not even thinking about the *.sso.o.o subdomain) 2020-04-28T18:49:21 oh we will 2020-04-28T18:49:35 frankly I am baffled novell chose this big of a subset 2020-04-28T18:50:38 well, we will add numbers after names then, that seems like a good solution to the problem 🤷‍♂️ 2020-04-28T18:51:37 we could also use xn--* encoding for the subdomain *g,d&r* 2020-04-28T18:53:17 that's not the worst of an idea, although communicating that to the user will be hard 2020-04-28T18:53:55 well, if someone was crazy enough to choose @#$%1* as username, why should only we suffer? ;-) 2020-04-28T18:54:52 oh yeah, I suffer looking at it 2020-04-28T18:55:25 do you have an idea how many users picked those funny chars in their username? 2020-04-28T18:55:54 well, you have access to wiki database for example, you could check there 2020-04-28T18:56:31 good point, I'll do that later (after the AppArmor meeting, and a video conference that will start in some minutes) 2020-04-28T18:56:53 yup 2020-04-28T18:58:08 I hope you have seen my mention of wiki plugin that enables openid connect login 2020-04-28T19:57:53 lcp: acl is_sso hdr_reg(host) -i .+\.sso\.opensuse\.org 2020-04-28T19:59:59 looks good to me 2020-04-28T20:00:16 whether is works or not, another matter ;) 2020-04-28T21:13:23 lcp: on opensuse-translation, there's a report that the new search.o.o supports less languages than the old www.o.o/searchPage 2020-04-28T21:14:00 is this something you can easily fix (for example, by copying over existing translations - assuming search.o. needs the same strings)? 2020-04-28T21:16:27 it doesn't 2020-04-28T21:16:58 I asked sbrabec for search-o-o to be included in weblate, I got no reply 2020-04-28T21:17:43 nice[tm] 2020-04-28T21:17:59 I'd say it's time to send a reminder ;-) 2020-04-28T21:18:48 maybe