2020-05-18T02:40:27  *** okurz_ is now known as okurz
2020-05-18T03:27:54  <bmwiedemann1> Eighth_Doctor: got this in apache error log: AH00037: Symbolic link not allowed or link target not accessible: /usr/share/ipsilon/ui/fonts, referer: https://www.opensuse.org/idp/ui/css/patternfly.css
2020-05-18T03:29:10  <bmwiedemann1> and somehow it still cannot access the ldaps, even though ldapsearch now works in CLI. Does the python code use a different CA store?
2020-05-18T03:32:30  <bmwiedemann1> This symlink is fixed by adding a  Options +FollowSymLinks for /usr/share/ipsilon
2020-05-18T03:35:13  <bmwiedemann1> ERROR: SERVER_DOWN: Can't contact LDAP server No such file or directory  -- strange error reason
2020-05-18T03:41:13  <Eighth_Doctor> bmwiedemann1: it shouldn't be using a different CA store
2020-05-18T03:41:26  <Eighth_Doctor> it picks up what is used by python itself
2020-05-18T03:41:38  <Eighth_Doctor> and mod_wsgi
2020-05-18T03:42:51  <bmwiedemann1> I just remember the SSL struggles with OpenStack and requests/urllib doing things differently
2020-05-18T03:43:33  <Eighth_Doctor> well, afaict, we don't do anything special in ipsilon
2020-05-18T03:43:55  <bmwiedemann1> but I cannot even find the "Can't contact" message in the code.
2020-05-18T03:44:52  <bmwiedemann1> Binary file /usr/lib64/libldap_r-2.4.so.2.10.9 matches
2020-05-18T03:45:23  <Eighth_Doctor> so our extent of ssl stuff is only when working with oidc/saml settings (setting validation)
2020-05-18T03:45:30  <Eighth_Doctor> and gssapi things
2020-05-18T03:45:37  <Eighth_Doctor> but otherwise, we don't do anything here
2020-05-18T03:46:38  <Eighth_Doctor> so that would be at the python3-ldap module
2020-05-18T03:47:07  <Eighth_Doctor> which means you're probably hitting libopenldap stuff
2020-05-18T03:47:15  <bmwiedemann1> and the LDAP server to use would be stored in adminconfig.sqlite DB?
2020-05-18T03:47:23  <Eighth_Doctor> should be
2020-05-18T03:47:34  <Eighth_Doctor> if you're using the sqlite as the database
2020-05-18T03:47:43  <Eighth_Doctor> though note, production deployments typically use postgres
2020-05-18T03:50:20  <bmwiedemann1> mariadb should also work?
2020-05-18T03:50:42  <Eighth_Doctor> never tested it personally
2020-05-18T03:50:48  <Eighth_Doctor> no idea
2020-05-18T03:50:56  <Eighth_Doctor> but feel free to try :)
2020-05-18T03:51:03  <bmwiedemann1> I think we already have a cluster running there and I dont want to run a postgres just for ipsilon
2020-05-18T04:03:30  <bmwiedemann1> OK, had a wrong server in login_config. Reached next level :-)
2020-05-18T04:06:14  <bmwiedemann1> and somehow I'm still logged in after "Log Out"
2020-05-18T04:19:21  <bmwiedemann1> OK, another minor issue are fonts. Did zypper in google-opensans-fonts fontawesome-fonts ; cd /usr/share/fonts ; ln -s truetype open-sans ; ln -s truetype fontawesome
2020-05-18T04:35:02  <bmwiedemann1> Eighth_Doctor: and this: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: nonce [SQL: 'DELETE FROM nonce WHERE nonce.uuid IN ...
2020-05-18T04:35:52  <Eighth_Doctor> bmwiedemann1: that's this issue: https://pagure.io/ipsilon/pull-request/306
2020-05-18T04:36:17  <Eighth_Doctor> it hasn't been merged yet because it's waiting for db migration code stuff, but that fixes that issue
2020-05-18T04:39:06  <bmwiedemann1> is it just me or does ipsilon have plenty rough edges?
2020-05-18T04:40:17  <Eighth_Doctor> bmwiedemann1: most of the roughness comes from the somewhat freshly done python3 port
2020-05-18T04:40:28  <Eighth_Doctor> some of it is also because you're doing a less commonly done configuration
2020-05-18T04:40:58  <Eighth_Doctor> and some of it is because I didn't finish adapting it to openSUSE distribution before you started working on it
2020-05-18T04:41:27  <bmwiedemann1> fair enough
2020-05-18T04:42:07  -heroes-bot- PROBLEM: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 200 * total waiting locks: 96 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-05-18T04:42:28  <Eighth_Doctor> bmwiedemann1: I do appreciate you doing this though
2020-05-18T04:42:43  <Eighth_Doctor> my goal is to have ipsilon 3.0.0 released after fedora and opensuse instances are live using the latest code
2020-05-18T04:43:26  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total waiting locks: 4 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-05-18T04:43:46  <bmwiedemann1> it seems, we do have a postgresql server as well :-)
2020-05-18T04:45:35  <bmwiedemann1> Eighth_Doctor: and I like ipsilon still way better than the openid-ldap code. Not only is it PHP, but also it is rather old and unmaintained
2020-05-18T04:45:52  <Eighth_Doctor> :D
2020-05-18T04:46:08  <Eighth_Doctor> and naturally, I welcome contributors to ipsilon :)
2020-05-18T04:46:51  <Eighth_Doctor> openSUSE will be the third public deployment of ipsilon, which makes me happy :)
2020-05-18T04:46:59  <Eighth_Doctor> (there are a few other private ones, but ehh)
2020-05-18T04:47:00  <bmwiedemann1> sooner or later you will get some one-liner patches from me, but I'm not a big python guru
2020-05-18T04:47:11  <Eighth_Doctor> that's okay :)
2020-05-18T04:47:45  <Eighth_Doctor> actually, fourth public deployment :P
2020-05-18T04:48:11  <Eighth_Doctor> Fedora, GNOME, RPM Fusion, and now openSUSE :D
2020-05-18T04:52:08  -heroes-bot- RECOVERY: PSQL locks on mirrordb2.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb2.infra.opensuse.org&service=PSQL%20locks
2020-05-18T04:53:26  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=12 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-05-18T04:57:47  <bmwiedemann1> need to prepare breakfast and get the kids to Kindergarten. See you.
2020-05-18T05:09:34  <Eighth_Doctor> I'm going to sleep now
2020-05-18T05:09:35  <Eighth_Doctor> bye :)
2020-05-18T05:57:58  <adrianS> moin
2020-05-18T06:09:39  * adrianS prepares the login server for switching
2020-05-18T06:31:14  <pjessen> morgen
2020-05-18T06:54:10  <adrianS> k, so I am going to change the DNS name for forums in next minutes .... it will take some time to propagate...
2020-05-18T06:58:06  <pjessen> okay
2020-05-18T06:59:53  <adrianS> and switching the other instances now to new authentification system ...
2020-05-18T07:01:04  <pjessen> looks good
2020-05-18T07:01:54  <adrianS> wiki works for me at least
2020-05-18T07:02:53  <pjessen> forums work for me
2020-05-18T07:03:05  <pjessen> well, i've logged in
2020-05-18T07:03:57  <adrianS> https://idp-portal-info.suse.com/   is updated
2020-05-18T07:04:04  <adrianS> marking forums as done there now as well
2020-05-18T07:05:39  <adrianS> that looks too easy :)
2020-05-18T07:06:12  <pjessen> yeah, much too easy
2020-05-18T07:15:08  * adrianS prepares some tea
2020-05-18T07:19:16  <pjessen> on articles I could read just 10mins ago, I am now getting "corrupted content error"
2020-05-18T07:19:48  <pjessen> "a network protocol violation that cannot be repaired"
2020-05-18T07:22:16  <adrianS> hm, is this a problem inside of the forums?
2020-05-18T07:22:21  <adrianS> do you have a link?
2020-05-18T07:22:34  <pjessen> heh, now it is working again
2020-05-18T07:30:07  * kl_eisbaer2 questions myself how to write this switch on status.opensuse.org. Is this an incident? ...a maintenance window? ...or just an improvement? :-)
2020-05-18T07:30:18  <kl_eisbaer2> THANK YOU, guys!
2020-05-18T07:37:20  <kl_eisbaer2> pjessen: are you writing an update to your original announcement Email ?
2020-05-18T08:19:05  <pjessen> kl_eisbaer2: I guess I should, yeah. I will.
2020-05-18T08:48:00  -heroes-bot- PROBLEM: NRPE on olaf.infra.opensuse.org - CHECK_NRPE: Error - Could not connect to 192.168.47.17: Connection reset by peer ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=olaf.infra.opensuse.org&service=NRPE
2020-05-18T08:57:59  -heroes-bot- RECOVERY: NRPE on olaf.infra.opensuse.org - NRPE v3.2.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=olaf.infra.opensuse.org&service=NRPE
2020-05-18T09:07:41  <lcp> adrianS: oh yeah, can we still get the email verified header in the proxy?
2020-05-18T09:17:40  <adrianS> lcp: most likely ... need to verify that the ldap attribute will still be used
2020-05-18T09:17:49  <adrianS> not that I only export old data ...
2020-05-18T09:21:50  <lcp> alright
2020-05-18T10:26:58  <bmwiedemann1> Eighth_Doctor: curl https://www.opensuse.org/idp/openid/XRDS contains the internal hostname instead of the proxy addr :-(
2020-05-18T11:16:20  <Eighth_Doctor> bmwiedemann1: when you initialized the ipsilon server, did you give it the name of the public hostname?
2020-05-18T11:19:46  <Eighth_Doctor> `ipsilon-server-install` has a `--hostname` parameter which you are supposed to set to the public FQDN
2020-05-18T11:20:02  <Eighth_Doctor> if you don't it'll try to figure it out itself
2020-05-18T11:25:36  <bmwiedemann1> ah, I didnt.
2020-05-18T11:25:59  <bmwiedemann1> can I also give it the proxy setting?
2020-05-18T11:26:28  <lcp> I don't think that's a switch ipsilon has
2020-05-18T11:26:42  <lcp> Conan Kudo: that would be a good thing to add
2020-05-18T11:28:40  <Eighth_Doctor> yeah that's currently not a switch
2020-05-18T11:28:57  <Eighth_Doctor> if someone captures that as an RFE on ipsilon's issue tracker, I'll look into it
2020-05-18T11:30:21  <bmwiedemann1> Now, it gets a lot further - testing with ci.opensuse.org : Failed to login: Error verifying signature with the OP: null
2020-05-18T11:31:35  <Eighth_Doctor> what is the public address supposed to be now?
2020-05-18T11:32:01  <Eighth_Doctor> because sso.opensuse.org just throws 503s
2020-05-18T11:32:16  <Eighth_Doctor> (which was the address LCP and I deployed ipsilon to)
2020-05-18T11:32:38  <Eighth_Doctor> is there a different address?
2020-05-18T11:32:40  <bmwiedemann1> for testing I use https://www.opensuse.org/idp/  but we might shift it later to be under the old /openid/user one
2020-05-18T11:33:36  <bmwiedemann1> logs show requests for /idp/openid/id/bmwiedemann.xrds and /idp/openid/yadis/bmwiedemann.xrds
2020-05-18T11:36:10  <Eighth_Doctor> hmm
2020-05-18T11:36:21  <lcp> Conan Kudo: that's because the vm run out of space
2020-05-18T11:36:46  <lcp> so it can't create new sessions
2020-05-18T11:36:49  <Eighth_Doctor> did we leave it in the huge debug mode?
2020-05-18T11:37:07  <lcp> prolly
2020-05-18T11:37:16  <Eighth_Doctor> 🤦‍♂️
2020-05-18T11:38:29  <lcp> the bigger issue is probably that the sessions aren't getting removed
2020-05-18T11:38:34  <lcp> and a session is created in every page access
2020-05-18T11:38:48  <bmwiedemann1> auto-expiry FTW
2020-05-18T11:39:30  <lcp> well, the easiest way to fix it would be to use pg for sessions instead of files
2020-05-18T11:39:48  <lcp> but sql sessions are currently broken and just don't work
2020-05-18T11:40:04  <lcp> tested it, didn't have the time to debug
2020-05-18T11:41:59  -heroes-bot- PROBLEM: NRPE on olaf.infra.opensuse.org - CHECK_NRPE: Error - Could not connect to 192.168.47.17: Connection reset by peer ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=olaf.infra.opensuse.org&service=NRPE
2020-05-18T12:22:39  <malcolmlewis> pjessen, hey seems forum email notifications are not working, any idea about that?
2020-05-18T12:27:49  <lcp> malcolmlewis: it might be because smtp port in vb is set to 26
2020-05-18T12:28:19  <lcp> I changed it to 25, let's see if it works now
2020-05-18T12:31:51  <malcolmlewis> lcp, thanks :)
2020-05-18T12:54:26  <malcolmlewis> lcp, email notifications all good now ;)
2020-05-18T12:55:18  <lcp> glad to hear it worked
2020-05-18T13:35:52  <bmwiedemann1> Eighth_Doctor: could it be that I need the nonce patch for openid to work?
2020-05-18T13:36:03  <Eighth_Doctor> bmwiedemann1: it's possible
2020-05-18T13:36:16  <Eighth_Doctor> it's applied on the instance LCP and I run as well
2020-05-18T13:36:46  <Eighth_Doctor> I've been a bit leery about it because it's not finished and doesn't have a db migration written in
2020-05-18T13:37:50  <pjessen> malcolmlewis: I'll check.
2020-05-18T13:38:11  <bmwiedemann1> https://pagure.io/ipsilon/pull-request/306#_1__42 looks like a migration, no?
2020-05-18T13:40:17  <pjessen> malcolmlewisL mails are being sent, just not many. I see two having gone out to xs4all.nl, for instance
2020-05-18T13:40:41  <Eighth_Doctor> bmwiedemann1: ah I missed that
2020-05-18T13:40:55  <Eighth_Doctor> I need to run tests and see if it's good
2020-05-18T13:47:27  -heroes-bot- PROBLEM: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS CRITICAL: DB postgres total locks: 220 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-05-18T13:49:32  <bmwiedemann1> Eighth_Doctor: openid still does not work :-( how does one best debug that?
2020-05-18T13:50:11  <Eighth_Doctor> you can turn on response debugging and look at the logs
2020-05-18T13:50:27  <Eighth_Doctor> err response logging
2020-05-18T13:50:39  <Eighth_Doctor> off the top of my head I can't remember the key, but LCP knows it
2020-05-18T13:50:55  * Eighth_Doctor is going into work now
2020-05-18T13:54:11  <bmwiedemann1> I used debug = True
2020-05-18T13:54:33  <bmwiedemann1> but nothing obvious/wrong visible there
2020-05-18T13:55:03  <malcolmlewis> pjessen, looks like lcp fixed it, was the port ;)
2020-05-18T13:55:44  <pjessen> malcolmlewis: okay.  I guess I forgot to change it.
2020-05-18T14:07:26  -heroes-bot- RECOVERY: PSQL locks on mirrordb1.infra.opensuse.org - POSTGRES_LOCKS OK: DB postgres total=47 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=mirrordb1.infra.opensuse.org&service=PSQL%20locks
2020-05-18T16:11:32  <knurpht> forums.o.o not reachable. nor the admincp. Not getting an email about incorrect login attempts.
2020-05-18T16:11:53  <knurpht> is this a known issue?
2020-05-18T16:28:46  <lcp> oh yeah
2020-05-18T16:28:50  <knurpht>  malcolmlewis: can you access them?
2020-05-18T16:29:16  <knurpht> lcp: any idea?
2020-05-18T16:29:43  <lcp> logging into the proxy does that
2020-05-18T16:29:58  <lcp> it might be bugged plugin, let me take a look
2020-05-18T16:30:53  <lcp> eh, I'm locked out for now, gotta wait a moment
2020-05-18T16:31:33  <lcp> you can use https://forums.opensuse.org/cmd/ICSLogout to not have to deal with white screen for now
2020-05-18T16:32:20  <lcp> after 15 minutes you should be able to login into admincp from there
2020-05-18T16:34:53  <lcp> actually, this might be the fault of the proxy
2020-05-18T16:36:21  <knurpht> I'll wait, it still says the password is wrong and I know it is not.
2020-05-18T16:37:33  <lcp> pjessen: eh, vb is being stupid, it assumes everybody accessing from behind the proxy is a malicious actor, because somebody is probably trying to break into admincp/modcp
2020-05-18T16:37:58  <lcp> so it doesn't matter if the password is right or wrong
2020-05-18T16:38:19  <lcp> actually, accessing vb from i.o.o might work
2020-05-18T16:39:55  <lcp> it almost does
2020-05-18T16:40:25  <lcp> yeah, vb behind a proxy is a mistake
2020-05-18T16:47:04  <lcp> okay, fixed it
2020-05-18T16:47:14  <lcp> I removed the ip address strikes entirely
2020-05-18T16:48:06  <lcp> that however means that it's very easy for an attacker to try to bruteforce their way into an admin/mod account since there is no limit on how many times they can try to enter a password
2020-05-18T16:51:12  <knurpht> Mmmm
2020-05-18T16:51:41  <knurpht> AFAICS that risk then always existed ....
2020-05-18T16:53:11  <lcp> well, previously you did have a strike on ip for 15 minutes after 5 failed attempts
2020-05-18T16:53:22  <lcp> (which is what was triggered by the proxy here)
2020-05-18T16:55:08  <lcp> if I found the part of code that triggered this, I could try to retool it to work with referrer system
2020-05-18T16:55:46  <lcp> I am also lazy, so I will just get through the stuff with discourse
2020-05-18T16:56:05  <lcp> and I would rather work with ruby than php ;)
2020-05-18T16:56:12  <bmwiedemann1> lcp: does the proxy add a X-Forwarded-For header with the original IP? vb could use that instead
2020-05-18T16:58:57  <bmwiedemann1> I think, it is not default-on in haproxy, but can be enabled with 1 line option
2020-05-18T17:01:58  <lcp> it might not
2020-05-18T17:02:25  <lcp> it should also set https header most likely (for $_SERVER['HTTPS'])
2020-05-18T17:02:36  <lcp> it doesn't do any of that
2020-05-18T17:10:15  <cboltz> forums.opensuse.org/snoop.php shows   X-Forwarded-For: (null), 77.21.255.45
2020-05-18T17:10:37  <lcp> right, so that works
2020-05-18T17:11:15  <cboltz> there's also   X-Forwarded-Proto: https   - but no HTTPS header (as discussed yesterday)
2020-05-18T17:21:50  <lcp> yeah
2020-05-18T17:32:13  <lcp> cboltz: the (null) might be the issue
2020-05-18T17:32:24  <lcp> because afaict it checks for this
2020-05-18T17:34:37  <cboltz> the interesting thing is that services behind login2.o.o go through two proxies - first login2.o.o (daffy{1,2]), then haproxy (on anna/elsa)
2020-05-18T17:35:01  <cboltz> I'd have to check the config of both to find out where the header gets set
2020-05-18T17:35:24  <lcp> huh, interesting
2020-05-18T17:36:10  <lcp> I implemented protocol switching using $_SERVER['HTTP_X_FORWARDED_PROTO'] btw
2020-05-18T17:36:14  <lcp> so that part works now
2020-05-18T17:36:35  <lcp> you can now login internally and externally thanks to that
2020-05-18T17:39:09  <cboltz>     SetenvIf  X-Forwarded-For  ",\s+([^,]+)"  clientip=$1   # from forum.i.o.o apache config - so even apache might do something to that header
2020-05-18T17:40:37  <lcp> well, that's an env, it can be used somewhere else, but I would rather remove the part where the proxy sets an empty header
2020-05-18T17:41:18  <cboltz> actually that pinted me to something:
2020-05-18T17:41:27  <cboltz> s/pinted/pointed/
2020-05-18T17:41:43  <cboltz> SetEnv HTTPS on    gives me $_SERVER['HTTPS'] == 'on' (tested locally)
2020-05-18T17:44:13  <lcp> right
2020-05-18T17:47:04  <knurpht> Seems to be OK now. Admincp now also shows the SSL is OK
2020-05-18T17:51:38  <lcp> cboltz: `SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on`
2020-05-18T17:51:43  <lcp> that worked
2020-05-18T18:33:11  <cboltz> :-)
2020-05-18T18:36:37  <lcp> I'm still clueless about proxy stuff
2020-05-18T19:15:59  -heroes-bot- PROBLEM: NRPE on olaf.infra.opensuse.org - CHECK_NRPE: Error - Could not connect to 192.168.47.17: Connection reset by peer ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=olaf.infra.opensuse.org&service=NRPE
2020-05-18T19:26:00  -heroes-bot- RECOVERY: NRPE on olaf.infra.opensuse.org - NRPE v3.2.1 ; See https://monitor.opensuse.org/icinga/cgi-bin/extinfo.cgi?type=2&host=olaf.infra.opensuse.org&service=NRPE
2020-05-18T19:59:32  <cboltz> lcp: what do you think about my idea to use   |   or   ?   as newline replacement in the machine list? An answer to this (whatever you answer) is the only thing that stops me from merging !401
2020-05-18T20:00:10  <lcp> I don't really know
2020-05-18T20:03:08  <cboltz> I'm also not sure (and don't have a problem with merging as-is)
2020-05-18T20:03:44  <cboltz> maybe hostusage and alias columns might benefit from a separator, look for example at the aliases for mirrordb3 in your screenshot
2020-05-18T21:52:03  <lcp> I fixed IP detection in vb, by adding a config option that causes proxy variables to trigger
2020-05-18T21:52:23  <lcp> so now the correct IP is associated with the user
2020-05-18T21:52:29  <lcp> and not the proxy
2020-05-18T21:52:37  <lcp> I also turn the strike thing back on
2020-05-18T21:56:25  <lcp> and I can see peoples' IPs are correctly set, so it should work