2020-05-13T01:49:26 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 180 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T01:59:27 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=16 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T02:45:18 *** okurz_ is now known as okurz 2020-05-13T04:47:26 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 13 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T04:52:05 -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 294 * total waiting locks: 143 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks 2020-05-13T05:02:06 -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=2 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks 2020-05-13T05:37:27 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=49 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T06:01:26 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 52 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T09:52:18 adrianS: I'm curious, since I heard a bunch of people complain about the login proxy not allowing passwords containing certain characters and 62 characters limit, what is the login proxy actually running? 2020-05-13T09:53:11 hm, where did you hear that? 2020-05-13T09:53:46 IIRC there is a limit inside of UCS atm which we need to report ... I am not aware of a limit in the proxy 2020-05-13T09:54:20 actually, I do not know if we have a limit there ... but given I don't know since > 10 years, I assume there is none :) 2020-05-13T09:54:30 I don't actually remember, it was also true with the previous system afaik, but I did bring it up when people had issues with logging in and it generally helped 2020-05-13T09:54:37 the proxy is mainly apache2 plus some modules 2020-05-13T09:54:47 yeah, that's what I expected 2020-05-13T09:55:15 I would assume mod_auth_ldap with some form overlay 2020-05-13T09:56:09 yes, authentification is done via the usual mod_*ldap modules 2020-05-13T09:56:09 err, apparently it's mod_authz_ldap, whatever 2020-05-13T09:56:18 * err, apparently it's mod_authnz_ldap, whatever 2020-05-13T09:56:32 mod_authnz_ldap and mod_ldap 2020-05-13T09:57:36 plus an own variant of auth_memcookie for the header rewrite 2020-05-13T09:57:47 something has to be wrong considering with both systems special characters just failed to work, I'm gonna look around maybe I find something 2020-05-13T09:58:15 eg ! ? 2020-05-13T09:58:28 or do you have any other example char? 2020-05-13T09:58:43 or any report somewhere? 2020-05-13T09:59:35 apparently & was problematic 2020-05-13T09:59:47 I do actually have some example passwords from users tho 2020-05-13T10:00:22 &rNQsH5EUH2LkRfWAP%dRMv@eENiryxc%o!EujQi@RvtDr#woQt!q^32tTP275oEYuswUgC6Ji$e9&sQVAC5!4jwdqzsjXszmwms23p4Qe$r7hoqFXB7YBF8h@aDt$U@ 2020-05-13T10:01:29 they all seem to work just fine in ucs too, they are perfectly usable 2020-05-13T10:03:02 just when it comes to the proxy, it hates it 2020-05-13T10:03:54 129 chars ... 2020-05-13T10:04:19 k, will test a bit after lunch 2020-05-13T10:04:40 but are these new passwords? 2020-05-13T10:04:51 I mean why is the problem surfacing now? 2020-05-13T10:05:53 no idea, I assume it's all a side effect of people generating new, more secure passwords for the new stuff 2020-05-13T10:06:13 whereas they wouldn't have let's say 10 years ago for the previous one 2020-05-13T11:59:02 cboltz: wrt boo#1134447 When did this happen? 2020-05-13T11:59:33 I noticed it yesterday 2020-05-13T11:59:47 but maybe it was broken a bit longer - I don't use osc daily 2020-05-13T12:00:19 cboltz: Ok. Because the change with the credentials_mgr_class was a while ago now 2020-05-13T12:01:51 to limit it a bit - last known-for-sure osc usage was around April 9th 2020-05-13T12:02:08 which version do you use? 2020-05-13T12:02:17 whatever is in tumbleweed 2020-05-13T12:02:47 Ok. And you migrated your community account since then, right? 2020-05-13T12:02:55 yes 2020-05-13T12:03:17 it's really just the missing credentials_mgr_class= line 2020-05-13T12:03:34 Then it is not an osc bug. credentials_mgr_class was a while ago now 2020-05-13T12:03:42 for the versions - /var/log/zypp/history says 2020-05-13T12:03:43 2020-03-15 22:00:00|install|osc|0.168.2-2.1 2020-05-13T12:03:47 Jupp 2020-05-13T12:03:49 credentials_mgr_class was a while ago now 2020-05-13T12:03:50 2020-04-24 01:00:27|install|osc|0.168.2-1.1 2020-05-13T12:04:01 args. sorry 2020-05-13T12:04:33 .osc_cookiejar is the one to blame. You can use osc without credentials_mgr_class= line 2020-05-13T12:05:00 I tried deleting it, but it didn't help 2020-05-13T12:05:05 but let me check again to be sure ;-) 2020-05-13T12:06:04 kl_eisbaer: aw, dang it, I forgot mailman3 machine also will need postgres 2020-05-13T12:06:26 and it took an error for me to realize the error of my ways 2020-05-13T12:07:06 I just removed my credentails_mgr_class= line and still can use osc without problems 2020-05-13T12:07:16 hmm, strange - I can't break it again 2020-05-13T12:07:41 no idea what was broken yesterday :-/ - I only know that with credentails_mgr_class= added it started to work 2020-05-13T12:08:46 I added compat code for oscrc files without the creds line. So that an update wouldn't break old installations 2020-05-13T12:09:04 and tested the hell out of it ;-) 2020-05-13T12:09:08 ;-) 2020-05-13T12:09:47 since I can't reproduce the problem anymore, let's assume it was a temporary issue with an unknown reason 2020-05-13T12:10:06 so nothing to fix for now 2020-05-13T12:10:19 what scares me even more... 2020-05-13T12:11:32 wait, I think I just reproduced it 2020-05-13T12:12:55 .osc_cookiechar reduced to just a comment: #LWP-Cookies-2.0 or completely deleted 2020-05-13T12:13:07 .oscrc from yesterday 2020-05-13T12:14:43 .oscrc: https://paste.opensuse.org/174b1d0c (password edited) 2020-05-13T12:16:42 after removing user and pass (and letting osc re-add it) it works again 2020-05-13T12:17:06 and it also works after removing credentials_mgr_class= 2020-05-13T12:17:43 but there is another difference - now it's user=cboltz instead of user = cboltz (and same spacing change for pass) 2020-05-13T12:24:38 That makes it even more weird. I cannot reproduce it at all 2020-05-13T12:25:36 I remove ~/.osc_cooloejar and have this oscrc: https://paste.opensuse.org/92308119 2020-05-13T12:25:41 And it just works 2020-05-13T12:26:06 even with the spaces and without the creds line 2020-05-13T12:34:47 damn, I'm afraid I finally found it - I was blind :-/ (and the whitespace diff helped to hide this) 2020-05-13T12:35:04 in the broken .oscrc there were two lowercase letters that need to be uppercase 2020-05-13T12:35:39 but I'm quite sure I use the same password on the new login server 2020-05-13T12:36:15 just wondering - could it be that the password was checked case-insensitive before? 2020-05-13T12:36:27 (and: sorry for wasting your time!) 2020-05-13T12:42:43 That could be :-) And you never waste my time ;-) 2020-05-13T12:49:46 :-) 2020-05-13T13:06:55 cboltz: god, I don't even want to think about what this means for the security 2020-05-13T13:41:12 adrianS: it is 128 chars. wc counts the newline, too 2020-05-13T13:45:17 is it just me or are the emails from progress seriously delayed 2020-05-13T14:05:55 bmwiedemann1: what is 128 chars? a password? 2020-05-13T14:06:24 the long password not working via OBS webui issue got just solved hopefully 2020-05-13T14:06:39 I could not find issues with special chars so far 2020-05-13T14:18:08 adrianS: what was it? 2020-05-13T14:18:40 a wrong base64 encoding in auth-up page ... inserting new lines 2020-05-13T14:18:53 seems eDirectory had a size limit for the password ... 2020-05-13T14:19:03 since it was there since beginning 2020-05-13T14:19:23 it also apparently accepts passwords regardless of capitalization, which makes this interesting 2020-05-13T14:19:40 eh, I am glad it's not gonna be a thing anymore 2020-05-13T17:35:35 lcp: I'm just looking at !399 and have some questions 2020-05-13T17:35:55 you changed the database_host from name to IP - why? ;-) 2020-05-13T17:36:00 huh? 2020-05-13T17:36:34 ah, because that ip isn't resolvable and kl_eisbaer advised to use the ip for postgres 2020-05-13T17:36:56 ah, ok 2020-05-13T17:37:13 nginx config 2020-05-13T17:37:22 well, not really a question about your config, more a general question 2020-05-13T17:37:40 we typically have "web_*" as role names for webservers 2020-05-13T17:37:56 and IIRC the nginx test only checks web_* roles 2020-05-13T17:38:27 so - do you think renaming the role wo web_mailman 3 would make sense, or should we drop the web_* name requirement in the test? 2020-05-13T17:38:32 right, I should probably also break up the matrix one then 2020-05-13T17:38:59 the question is if we should keep the web_* prefix as hard requirement 2020-05-13T17:39:18 it makes sense for services that are "mostly a webserver" (like web_static) 2020-05-13T17:39:33 or web_jekyll 2020-05-13T17:39:39 I don't know tbh 2020-05-13T17:40:21 but I'm not sure if it makes sense for matrix which is (at least that's my understanding, I only know it from your salt files) "lots of things + a side job web frontend" ;-) 2020-05-13T17:41:49 I tend to change the test so that it checks the nginx config in all roles 2020-05-13T17:42:20 (MR will follow later, and I'll keep it open for a week so that everybody has a chance to object ;-) 2020-05-13T17:43:04 next question: all the /var/log/mailman/* log files 2020-05-13T17:43:27 it looks like they all have the same settings, only different filenames 2020-05-13T17:43:55 well, they provide different logs, for every purpose 2020-05-13T17:44:07 right 2020-05-13T17:44:31 but IMHO they are perfect candidates for a {% for file in ['uwsgi.log', 'uwsgi-error.log', .... ] %} loop to make the salt code shorter and easier to understand 2020-05-13T17:45:39 yeah, you are probably right 2020-05-13T17:46:25 let's just say I have some "experience" with cleaning up duplicated code and know how much pain this can cause ;-) 2020-05-13T19:48:07 lcp, db extract has been finished, woohoo!! 2020-05-13T19:49:25 malcolmlewis: I saw 2020-05-13T19:50:51 :-) 2020-05-13T19:52:30 we do have login setup already, so it's just a matter of per getting the db imported to vb 2020-05-13T20:15:51 lcp, yes AFAIK he's onto it as soon as he has time, I expect sometime Friday it will be done.... 2020-05-13T20:15:58 knurpht, ^^ 2020-05-13T20:21:24 malcolmlewis: thanks, let's hope so. 2020-05-13T20:39:26 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 61 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T20:49:27 -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=47 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T21:03:26 -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 62 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks 2020-05-13T22:27:31 *** nikki[m]1 is now known as nikki6669[m]